Neural text-to-speech synthesis using multi-level contextual features

Abstract

The present disclosure provides a method and apparatus for generating speech through neural text-to-speech (TTS) synthesis. A text input may be obtained. Phoneme features of the text input may be generated. Context features of the text input may be generated based on a set of sentences associated with the text input. A speech waveform corresponding to the text input may be generated based on the phoneme features and the context features.

Description

Background Art

[0001] Text-to-speech (TTS) synthesis aims to generate corresponding speech waveforms based on text input. TTS synthesis is widely used for speech-to-speech translation, voice customization for specific users, role-playing in stories, etc. Neural TTS systems are increasingly used to implement TTS synthesis, and are becoming a research hotspot in the field of artificial intelligence (AI) in recent years. Neural TTS systems can predict acoustic features based on text input, and then generate speech waveforms based on the predicted acoustic features. Unlike traditional TTS technology that requires well-designed front-end language features, neural TTS systems use an end-to-end structure to build models and can be trained directly based on, for example, text-speech data pairs. Neural TTS systems can jointly optimize the pronunciation, prosody, etc. of speech, which will result in more natural synthesized speech than traditional TTS technology. Summary of the invention

[0002] This summary is provided to introduce a set of concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure provide a method and apparatus for generating speech through neural TTS synthesis. A text input may be obtained. Phoneme features of the text input may be generated. Context features of the text input may be generated based on a set of sentences associated with the text input. A speech waveform corresponding to the text input may be generated based on the phoneme features and the context features.

[0004] It should be noted that one or more of the above aspects include the features specifically pointed out in the following detailed description and claims. The following description and drawings set forth in detail certain illustrative features of the one or more aspects. These features are merely indicative of the various ways in which the principles of various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0006] Figure 1 An exemplary conventional neural TTS system is shown.

[0007] Figure 2 An exemplary neural TTS system according to an embodiment of the present disclosure is shown.

[0008] Figure 3 is a schematic diagram showing context features according to an embodiment of the present disclosure.

[0009] Figure 4 An exemplary process for generating current semantic features according to an embodiment of the present disclosure is shown.

[0010] Figure 5 An exemplary process for generating historical acoustic features according to an embodiment of the present disclosure is shown.

[0011] Figure 6 An exemplary implementation of an acoustic encoder according to an embodiment of the present disclosure is shown.

[0012] Figure 7 An exemplary process for generating historical semantic features according to an embodiment of the present disclosure is shown.

[0013] Figure 8 An exemplary process for generating future semantic features according to an embodiment of the present disclosure is shown.

[0014] Fig. 9 An exemplary process for generating paragraph semantic features according to an embodiment of the present disclosure is shown.

[0015] Fig.10 An exemplary process for generating location features according to an embodiment of the present disclosure is shown.

[0016] Fig.11 Another exemplary neural TTS system according to an embodiment of the present disclosure is shown.

[0017] Fig.12 Yet another exemplary neural TTS system according to an embodiment of the present disclosure is shown.

[0018] Fig.13 is a flowchart of an exemplary method for generating speech through neural TTS synthesis according to an embodiment of the present disclosure.

[0019] Fig.14 An exemplary process for generating training data for a neural TTS system according to an embodiment of the present disclosure is shown.

[0020] Fig.15 An exemplary process for training a neural TTS system according to an embodiment of the present disclosure is shown.

[0021] Fig.16 is a flowchart of an exemplary method for generating speech through neural TTS synthesis according to an embodiment of the present disclosure.

[0022] Fig.17 An exemplary apparatus for generating speech through neural TTS synthesis according to an embodiment of the present disclosure is shown.

[0023] Fig.18 An exemplary apparatus for generating speech through neural TTS synthesis according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The present disclosure will now be discussed with reference to various exemplary embodiments. It should be understood that the discussion of these embodiments is only for enabling those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0025] A neural TTS system can be used to generate speech corresponding to a text input. A traditional neural TTS system generates speech using only phoneme features or character features of the current text input. The current text input is, for example, the current sentence. In this article, a phoneme feature refers to an information representation of a phoneme that pronounces the text input, which is generated based on a phoneme sequence recognized from the text input, wherein the phoneme sequence is a list of sequential phonemes that form the pronunciation of the text input. A character feature refers to an information representation of the characters that constitute the text input, which is generated based on a character sequence recognized from the text input, wherein the character sequence is a list of sequential characters contained in the text input.

[0026] Typically, the same text can correspond to different pronunciations, such as pronunciations with different speaking speeds, pitches, prosody, emotions, or pleasantness, etc. From a mathematical point of view, the conversion from text to speech can be viewed as a large-scale inverse problem, which decompresses a highly compressed source, such as text, into an extremely complex target, such as an audio signal. Neural TTS systems attempt to use machine learning techniques to solve this problem, which can be viewed as a one-to-many mapping problem from a machine learning perspective. Because traditional neural TTS systems only use the phoneme features or character features of the current text input to generate speech, lacking context-related information contained in the text input or the text input's neighboring text, traditional neural TTS systems cannot effectively solve the one-to-many mapping problem. Therefore, traditional neural TTS systems typically generate less expressive speech with fixed patterns, such as reading-style speech with flat prosody and emotion.

[0027] In addition, when generating speech for a group of sentences, a traditional neural TTS system usually first generates a corresponding speech waveform for each sentence in the group of sentences, and then combines all the generated speech waveforms to obtain a speech waveform corresponding to the group of sentences. A group of sentences is, for example, a paragraph. In addition, when combining, the pause duration between each speech waveform corresponding to each sentence is usually set to be the same or fixed. This will also lead to similar rhythms between sentences, making the generated speech boring.

[0028] Embodiments of the present disclosure propose to further use multi-level context features to improve the speech generation capability of the neural TTS system. Context features can be generated based on multiple levels of context information, such as word sequences of text input, word sequences and acoustic features of text input adjacent texts, position information of text input, etc. Acoustic features can include various traditional TTS acoustic features, such as mel-spectrum, linear spectral pair (LSP), etc. Position information can refer to information representation of the position of the text input in the adjacent text.

[0029] In one aspect, embodiments of the present disclosure propose to use both the phoneme features of the text input and the multi-level context features to generate speech corresponding to the text input. The context features generated based on the context information may contain prior knowledge about semantics and acoustics. The neural TTS system can learn common patterns of speech rate, pitch, prosody, emotion or pleasure, etc. from such prior knowledge. Therefore, when generating speech, considering more features such as the above-mentioned context features can help alleviate the one-to-many mapping problem and enhance the speech generation capability of the neural TTS system, thereby generating more natural and expressive speech.

[0030] In another aspect, the embodiments of the present disclosure propose that when generating a speech waveform for a group of sentences, the pauses between sentences can be modeled and the pause length can be determined according to context features. A group of sentences is, for example, a paragraph. Since the pause length between sentences is related to the context features, it can vary with different context features, so the rhythm of the speech finally generated will be richer and more natural.

[0031] Figure 1 An exemplary conventional neural TTS system 100 is shown. The neural TTS system 100 may be configured to generate a speech waveform 170 corresponding to a text input 102. The text input 102 may be a sentence containing one or more words, one or more phrases, and the like.

[0032] like Figure 1 As shown, a phoneme sequence 104 can be identified from a text input 102 by various existing techniques, such as Letter-to-Sound (LTS). The identified phoneme sequence 104 can be converted into a phoneme embedding vector sequence by a phoneme embedding model 106. The phoneme embedding vector sequence includes a set of phoneme embedding vectors, each phoneme embedding vector corresponding to a phoneme in the phoneme sequence 104.

[0033] The phoneme embedding vector sequence can be provided as an input to the encoder 110 of the neural TTS system 100. The encoder 110 can be based on various network structures. As an example, the encoder 110 may include one or more convolutional layers 112 and at least one bidirectional long short-term memory (BLSTM) layer 114. The encoder 110 can convert the information contained in the phoneme embedding vector sequence into a vector space that is more robust and more suitable for learning the alignment of the acoustic features output by the decoder. For example, the encoder 110 can convert the phoneme embedding vector sequence into a phoneme feature 120 in the vector space. In this article, the feature corresponding to the phoneme sequence of the text input generated by the encoder is referred to as a phoneme feature. It should be understood that in other embodiments, the phoneme embedding model 106 for converting the phoneme sequence into a phoneme embedding vector sequence can also be trained or updated in association with the neural TTS system 100. In this case, the phoneme embedding model 106 can be located inside the neural TTS system 100, for example, inside the encoder 110.

[0034] It should be understood that, as an alternative to identifying a phoneme sequence, a character sequence can also be identified from the text input 102, and the identified character sequence can be further converted into a character embedding vector sequence. The character embedding vector sequence can also be provided as an input to the encoder 110, thereby generating a character feature based on the character sequence corresponding to the text input 102.

[0035] The neural TTS system 100 may include an attention unit 130. The attention unit 130 may implement an attention mechanism that serves as a bridge between the encoder 110 and the decoder 140. For example, the attention mechanism may help align the phoneme features output by the encoder 110 with the acoustic features 150 output by the decoder. The attention unit 130 may implement various types of attention mechanisms, such as soft attention, hard attention, position-sensitive attention, Gaussian mixture model (GMM) attention, and the like.

[0036] The decoder 140 may include a pre-net 142 consisting of a feed-forward layer, a long short-term memory (LSTM) 144, a linear map 146, a post-net 148 consisting of a convolutional layer, etc. The LSTM 144 may receive input from the pre-net 142 and provide its output to the linear map 146, while the processing of the LSTM 144 is affected by the attention unit 130. The linear map 146 may provide its output to the pre-net 142 and the post-net 148, respectively. Finally, the output of the post-net 148 is combined with the output of the linear map 146 to generate an acoustic feature 150. In one embodiment, the linear map 146 may also be used to generate a stop sign.

[0037] The neural TTS system 100 may further include a vocoder 160. The vocoder 160 may generate a speech waveform 170 based on the acoustic features 150 output by the decoder 140. The vocoder 160 may be based on various network structures, such as a Wavenet vocoder, a Griffin-Lim vocoder, and the like.

[0038] Figure 1 The conventional neural TTS system 100 in the present invention generates a speech waveform corresponding to the text input based only on the phoneme features corresponding to the phoneme sequence of the text input. In order to enhance the naturalness of the generated speech, such as speech rate, pitch, rhythm, emotion or pleasure, the embodiments of the present disclosure propose to use both the phoneme features and context features of the text input to generate a speech waveform corresponding to the text input. Figure 1 Compared with the traditional neural TTS system 100 in the neural TTS system, the embodiments of the present disclosure improve at least the input part of the decoder in the neural TTS system. Figure 2 An exemplary neural TTS system 200 is shown in accordance with an embodiment of the present disclosure.

[0039] like Figure 2 As shown, a phoneme sequence 204 can be identified from a text input 202. The identified phoneme sequence 204 can be converted into a phoneme embedding vector sequence by a phoneme embedding model 206. The phoneme embedding model 206 can correspond to Figure 1 The phoneme embedding model 106 in . The phoneme embedding vector sequence can be provided as input to the encoder 208 of the neural TTS system 200. The encoder 208 can correspond to Figure 1 The encoder 110 in . The encoder 208 can generate a phoneme feature 210 based on the phoneme embedding vector sequence.

[0040] In addition to generating the phoneme features 210 of the text input 202, the neural TTS system 200 according to the embodiment of the present disclosure can also generate context features 218 of the text input 202. A set of sentences associated with the text input 202 can be obtained, such as the paragraph 212 where the text input 202 is located. Context information 214 can be extracted from the paragraph 212, such as the word sequence of the text input 202, the word sequence and acoustic features of the text adjacent to the text input 202, the position information of the text input 202, etc. The context features 218 can be generated based on the context information 214 by the feature generation unit 216. For different context information, the feature generation unit 216 can have different structures. For example, when the context information 214 is a word sequence of the text input 202, the feature generation unit 216 can include a word embedding model, an upsampling unit, an encoder, etc. The feature generation unit 216 can be located entirely inside the neural TTS system 200, or only a portion of it can be located inside the neural TTS system 200.

[0041] The generated phoneme features 210 and context features 218 may be combined into a mixed feature by a cascade unit 220. The attention unit 222 may implement an attention mechanism, such as a position-sensitive attention mechanism, on the mixed feature. The mixed feature to which the attention mechanism is implemented may be provided to a decoder 224. The decoder 224 may correspond to Figure 1 The decoder 224 may generate an acoustic feature corresponding to the mixed feature based on the mixed feature to which the attention mechanism is implemented. The acoustic feature may then be provided to the vocoder 226. The vocoder 226 may correspond to Figure 1 The vocoder 160 in . The vocoder 226 can generate a speech waveform corresponding to the text input 202.

[0042] Figure 3 is a schematic diagram showing a context feature 300 according to an embodiment of the present disclosure. The context feature 300 may correspond to Figure 2 Context features 218 in the example, which may include multiple levels of features.

[0043] The context features 300 may include current semantic features 310. In this article, the current semantic features refer to features generated based on a word sequence of a text input. The text input is, for example, a current sentence. The current semantic features may reflect or include semantic information of the current sentence. Figure 4 To illustrate the specific process of generating the current semantic features.

[0044] The context features 300 may also include global features 320. The global features 320 may include historical and future context features of the text input, such as historical acoustic features 322, historical semantic features 324, future semantic features 326, paragraph semantic features 328, and position features 330. In this article, historical acoustic features refer to features generated based on acoustic features of previous sentences of the text input, which may reflect or contain acoustic information related to the speaker's expression and acoustic state when speaking the previous sentence. Figure 5 To illustrate the specific process of generating historical acoustic features. Historical semantic features refer to features generated based on the word sequence of the previous segment of the text input, which can reflect or contain the semantic information of the previous segment. Figure 7 To illustrate the specific process of generating historical semantic features. Future semantic features refer to features generated based on the word sequence of subsequent segments of text input, which can reflect or contain the semantic information of subsequent segments. Figure 8 To illustrate the specific process of generating future semantic features. Paragraph semantic features refer to features generated based on the word sequence of the paragraph in which the text input is located. They can reflect or contain the semantic information of the paragraph, such as the central idea of ​​the paragraph. Fig. 9To illustrate the specific process of generating paragraph semantic features. Position features refer to features generated based on the position information of text input in a paragraph, which can reflect the position of text input in a paragraph. Fig.10 To illustrate the specific process of generating paragraph semantic features.

[0045] Return to reference Figure 2 , the phoneme features and context features of the text input are cascaded and provided to the attention unit and the decoder in sequence. It should be understood that Figure 2 The context features 218 may include reference Figure 3 Any one or more of the described context features. For example, the context features 218 may include only current semantic features, historical acoustic features, and historical semantic features.

[0046] Figure 4 An exemplary process 400 for generating a current semantic feature according to an embodiment of the present disclosure is shown. A word sequence 404 can be identified from a text input 402. The word sequence 404 is a list of sequential words that make up the text input 402.

[0047] The current semantic feature 420 may be generated based on the word sequence 404 by the feature generation unit 410. The feature generation unit 410 may correspond to Figure 2 The feature generation unit 216 in . The feature generation unit 410 may include a word embedding model 412, an upsampling unit 414 and an encoder 416.

[0048] Word embedding model 412 can be based on natural language processing (NLP) technology, such as neural machine translation (NMT). Both word embedding model and neural TTS system have similar sequence to sequence encoder-decoder framework, thereby facilitating network aggregation. In one embodiment, bidirectional encoder representation (Bidirectional Encoder Resentations from Transformers, BERT) model from converter can be adopted as word embedding model. Word embedding vector sequence can be generated based on word sequence 404 by word embedding model 412, wherein each word has corresponding embedding vector, and all these embedding vectors form word embedding vector sequence. Word embedding vector contains the meaning and semantic context information of word, which will help to improve the naturalness of generated speech. In addition, word embedding vector will also help to solve the word segmentation problem generated for speech of Chinese text.

[0049] The word embedding vector sequence can be upsampled by upsampling unit 414 to align with the phoneme sequence of text input 402. For example, a word can be pronounced using one or more phonemes. Therefore, during upsampling, each word embedding vector in the word embedding vector sequence can be repeated a number of times corresponding to the number of phonemes in the word. The upsampled word embedding vector sequence can be provided to encoder 416. Encoder 416 can have Figure 1 The encoder 416 may have a network structure similar to the encoder 110 in FIG. 4 , for example, including one or more convolutional layers and at least one BLSTM layer. The encoder 416 may generate a current semantic feature 420 based on the upsampled word embedding vector sequence.

[0050] Figure 5 An exemplary process 500 for generating historical acoustic features according to an embodiment of the present disclosure is shown. Acoustic features corresponding to at least one previous sentence preceding the text input may be obtained. In the training phase, the real acoustic features of the previous sentence may be used. In the prediction phase, the predicted acoustic features of the previous sentence may be used. The acoustic features may include various traditional TTS acoustic features, such as Mel-spectrogram, LSP, etc. Assuming that the text input is sentence i, acoustic features 502-1, 502-2, ... 502-k corresponding to sentence i-1, sentence i-2, ... sentence ik, respectively, may be obtained. In one embodiment, acoustic features corresponding to two sentences preceding the text input may be obtained, i.e., k=2. Historical acoustic features 550 may be generated based on the acoustic features 502-1, 502-2, ... 502-k by a feature generation unit 510. The feature generation unit 510 may correspond to Figure 2 The feature generation unit 216 in the feature generation unit 510 may include acoustic encoders 512-1, 512-2, ... 512-k and up-sampling units 514-1, 514-2, ... 514-k corresponding to the acoustic features 502-1, 502-2, ... 502-k, respectively, and a cascade unit 516.

[0051] The corresponding acoustic features 502-1, 502-2, ... 502-k can be converted into speaker embedding vector sequences for sentence i-1, sentence i-2, ... sentence ik respectively through acoustic encoders 512-1, 512-2, ... 512-k. Figure 6514-k, respectively, to align with the phoneme sequence of sentence i. For example, the speaker embedding vector sequence of each sentence may be repeated a number of times corresponding to the number of phonemes of sentence i. The upsampled speaker embedding vector sequences may be combined into historical acoustic features 520 by a cascade unit 516.

[0052] Figure 6 FIG. 6 shows an exemplary implementation of an acoustic encoder 600 according to an embodiment of the present disclosure. The acoustic encoder 600 may correspond to Figure 5 Any one of the acoustic encoders 512-1, 512-2, ... 512-k in FIG. Figure 6 As shown, the acoustic encoder 600 can be based on a neural network, which is used to generate a speaker embedding vector sequence 612 for sentence m based on the acoustic features 602 of sentence m. The acoustic encoder 600 can include multiple convolutional layers 604, average pooling layers 606, multiple fully connected (FC) layers 608 and a mapping layer 610 in sequence. The acoustic encoder 600 can extract acoustic information related to the speaker's expression and acoustic state when speaking the previous sentence from the acoustic features of the previous sentence. Usually, the expression and acoustic state adopted by the speaker when speaking are inertial, that is, there is continuity in physical properties. When generating a speech waveform corresponding to the current sentence, considering the acoustic features of the previous sentence can help enhance the naturalness of the speech waveform of the current sentence.

[0053] Figure 7 An exemplary process 700 for generating historical semantic features according to an embodiment of the present disclosure is shown. A previous segment 702 located before the text input can be obtained. The previous segment 702 can include, for example, one or more sentences located before the text input. A word sequence 704 can be identified from the previous segment 702.

[0054] The historical semantic feature 730 may be generated based on the word sequence 704 by the feature generation unit 710. The feature generation unit 710 may correspond to Figure 2 The feature generation unit 216 in . The feature generation unit 710 may include a word embedding model 712, an average pooling layer 714, an upsampling unit 716, a dense layer 718 and an encoder 720.

[0055] The word embedding model 712 can generate a sequence of word embedding vectors based on the word sequence 704. The word embedding vector sequence can be average pooled by an average pooling layer 714 to generate a sequence of average segment embedding vectors. The average segment embedding vector sequence can be upsampled by an upsampling unit 716 to align with the phoneme sequence of the text input. For example, the average segment embedding vector sequence can be repeated a number of times corresponding to the number of phonemes in the text input. Then, a compressed representation of the average segment embedding vector sequence can be obtained by a dense layer 718. The compressed representation can be provided to an encoder 720 to generate historical semantic features 730. The encoder 720 can have the same Figure 1 The encoder 110 in FIG. 1 may have a network structure similar to that of the encoder 110, for example including one or more convolutional layers and at least one BLSTM layer.

[0056] In one embodiment, the previous segment 702 may include several complete sentences. For example, the number of sentences included in the previous segment 702 may be determined based on the number of characters that the word embedding model 712 can process. Taking the BERT model as a word embedding model as an example, the number of characters that the BERT model can process is 512, and one or more sentences with a total number of characters not exceeding 512 located before the text input may be selected as the previous segment 702.

[0057] Figure 8 An exemplary process 800 for generating future semantic features according to an embodiment of the present disclosure is shown. A subsequent segment 802 located after the text input can be obtained. The subsequent segment 802 can include, for example, one or more sentences located after the text input. A word sequence 804 can be identified from the subsequent segment 802.

[0058] The future semantic feature 830 may be generated based on the word sequence 804 by the feature generation unit 810. The feature generation unit 810 may correspond to Figure 2 The feature generation unit 216 in . The feature generation unit 810 may include a word embedding model 812, an average pooling layer 814, an upsampling unit 816, a dense layer 818 and an encoder 820.

[0059] The word embedding model 812 can generate a sequence of word embedding vectors based on the word sequence 804. The word embedding vector sequence can be average pooled by an average pooling layer 814 to generate a sequence of average segment embedding vectors. The average segment embedding vector sequence can be upsampled by an upsampling unit 816 to align with the phoneme sequence of the text input. For example, the average segment embedding vector sequence can be repeated a number of times corresponding to the number of phonemes in the text input. Then, a compressed representation of the average segment embedding vector sequence can be obtained by a dense layer 818. The compressed representation can be provided to an encoder 820 to generate future semantic features 830. The encoder 820 can have the same Figure 1 The encoder 110 in FIG. 1 may have a network structure similar to that of the encoder 110, for example including one or more convolutional layers and at least one BLSTM layer.

[0060] In one embodiment, the subsequent segment 802 may include several complete sentences. Similar to the previous segment 702, the number of sentences included in the subsequent segment 802 may be determined, for example, based on the number of characters that the word embedding model 812 can process. In another embodiment, the subsequent segment 802 may include only one sentence, that is, the sentence immediately following the text input.

[0061] Fig. 9 An exemplary process 900 for generating paragraph semantic features according to an embodiment of the present disclosure is shown. A paragraph text 902 of a paragraph in which a text input is located can be obtained. In one embodiment, a word sequence 904 can be directly identified from the paragraph text 902. In another embodiment, a central sentence 906 can be first extracted from the paragraph text 902, and then a word sequence 904 can be identified from the central sentence 906. The central sentence 906 can be extracted from the paragraph text 902 by various techniques, such as using a context-based summary (CRSums) model.

[0062] The paragraph semantic feature 930 may be generated based on the word sequence 904 by the feature generation unit 910. The feature generation unit 910 may correspond to Figure 2 The feature generation unit 216 in . The feature generation unit 910 may include a word embedding model 912, an average pooling layer 914, an upsampling unit 916, a dense layer 918 and an encoder 920.

[0063] The word embedding model 912 can generate a sequence of word embedding vectors based on the word sequence 904. The word embedding vector sequence can be average pooled by an average pooling layer 914 to generate an average paragraph embedding vector sequence. The average paragraph embedding vector sequence can be upsampled by an upsampling unit 916 to align with the phoneme sequence of the text input. For example, the average paragraph embedding vector sequence can be repeated a number of times corresponding to the number of phonemes in the text input. Then, a compressed representation of the average paragraph embedding vector sequence can be obtained by a dense layer 918. The compressed representation can be provided to an encoder 920 to generate a paragraph semantic feature 930. The encoder 920 can have the same Figure 1 The encoder 110 in FIG. 1 may have a network structure similar to that of the encoder 110, for example including one or more convolutional layers and at least one BLSTM layer.

[0064] Fig.10 An exemplary process 1000 for generating position features according to an embodiment of the present disclosure is shown. Position information 1002 of the text input in the paragraph can be extracted, such as information about the number of sentences the text input is from the beginning of the paragraph, information about the number of sentences the text input is from the end of the paragraph, etc.

[0065] The position feature 1020 may be generated based on the position information 1002 by the feature generation unit 1010. The feature generation unit 1010 may correspond to Figure 2 The feature generation unit 216 in the embodiment of the present invention can include a position embedding model 1012, an upsampling unit 1014, a dense layer 1016 and an encoder 1018.

[0066] The position embedding model 1012 can generate a position embedding vector sequence based on the position information 1002. The position embedding vector sequence can be upsampled by an upsampling unit 1014 to align with the phoneme sequence of the text input. For example, the position embedding vector sequence can be repeated a number of times corresponding to the number of phonemes in the text input. Then, through a dense layer 1016, a compressed representation of the upsampled position embedding vector sequence can be obtained. The compressed representation can be provided to an encoder 1018 to generate a position feature 1020. The encoder 1018 can have a position feature 1020 that is consistent with the position feature 1020. Figure 1 The encoder 110 in FIG. 1 may have a network structure similar to that of the encoder 110, for example including one or more convolutional layers and at least one BLSTM layer.

[0067] The position feature generated based on the position information can reflect the position of the text input in the paragraph. Generally, the position of the text input in the paragraph may affect the pitch of the speech corresponding to the text input. For example, when the text input is at the beginning of the paragraph, its pitch tends to be higher; when the text input is in the middle of the paragraph, its pitch tends to be flatter; and when the text input is at the end of the paragraph, its pitch will also be higher. Therefore, when generating the speech of the text input, considering the position information of the text input helps to enhance the naturalness of the generated speech.

[0068] It should be understood that Figure 4-Figure 5 , Figure 7-Figure 10 The processes 400-500, 700-1000 in the embodiment are merely examples of processes for generating various context features. According to actual application requirements, the process for generating context features may include any other steps, and may include more or fewer steps. For example, although in Figure 7-Figure 10 In processes 700-1000 in FIG. 1 , a dense layer is used to obtain a compressed representation of the embedded vector sequence and the compressed representation is provided to the encoder. However, in some embodiments, the embedded vector sequence can be directly provided to the encoder without being processed by the dense layer.

[0069] Return to reference Figure 2 , the context features are provided to the attention unit and the decoder only after being concatenated with the phoneme features. In order to strengthen the influence of the context features on the decoder, especially the influence of the global features on the decoder, the embodiments of the present disclosure also propose to implement the attention mechanism for the global features separately, and directly apply the global features with attention to the decoder. Fig.11 Another exemplary neural TTS system 1100 according to an embodiment of the disclosure is shown.

[0070] like Fig.11 As shown, the phoneme feature 1102 of the text input and the context feature 1104 can be combined into a first mixed feature through a cascade unit 1106. The phoneme feature 1102 can correspond to Figure 1 The phoneme features 120 in . The contextual features 1104 may correspond to Figure 3 The context features 300 in the first mixed feature include, for example, current semantic features and global features, wherein the global features may include at least one of historical acoustic features, historical semantic features, future semantic features, paragraph semantic features, and position features, etc. The attention unit 1108 may implement an attention mechanism, such as a position-sensitive attention mechanism, on the first mixed feature.

[0071] In addition, the attention mechanism can be implemented separately for the global feature 1110. For example, the global feature 1110 can be provided to the attention unit 1112. The global feature 1110 can be used with Figure 3The global feature 320 in the example includes at least one of historical acoustic features, historical semantic features, future semantic features, paragraph semantic features, and position features. The attention unit 1112 may implement an attention mechanism, such as a position-sensitive attention mechanism, on the global feature 1110. The global feature implemented with attention may be cascaded 1114 with the first mixed feature implemented with attention, thereby obtaining a second mixed feature. The second mixed feature may be provided to a decoder 1116. The decoder 1116 may correspond to Figure 1 The decoder 1116 may generate an acoustic feature corresponding to the second mixed feature. The acoustic feature may then be provided to the vocoder 1118 to generate a speech waveform corresponding to the text input.

[0072] It should be understood that Fig.11 The context features 1104 may include reference Figure 3 Some or all of the described context features, for example, context feature 1104 may include only current semantic features, historical acoustic features, and historical semantic features. Fig.11 The global features 1110 in may include reference Figure 3 Some or all of the described global features, such as global features 1110 , may include only historical acoustic features and historical semantic features.

[0073] Fig.12 FIG. 1 shows another exemplary neural TTS system 1200 according to an embodiment of the present disclosure. Fig.12 As shown, the phoneme feature 1202 of the text input and the context feature 1204 can be combined into a first mixed feature through a cascade unit 1206. The phoneme feature 1202 can correspond to Figure 1 The phoneme features 120 in . The contextual features 1204 may correspond to Figure 3 The context features 300 in the first mixed feature include, for example, current semantic features and global features, wherein the global features may include at least one of historical acoustic features, historical semantic features, future semantic features, paragraph semantic features, and position features, etc. The attention unit 1208 may implement an attention mechanism, such as a position-sensitive attention mechanism, on the first mixed feature.

[0074] In addition, average pooling 1212 may be performed on the global features 1210 to obtain an average global feature. The global features 1210 may be Figure 3The average global feature may correspond to the global feature 320 in the image, for example, including at least one of historical acoustic features, historical semantic features, future semantic features, paragraph semantic features, and position features. The average global feature may be cascaded 1214 with the first mixed feature implemented with attention, thereby obtaining a second mixed feature. The second mixed feature may be provided to a decoder 1216. The decoder 1216 may correspond to Figure 1 The decoder 140 in . The decoder 1216 can generate an acoustic feature corresponding to the second mixed feature based on the second mixed feature. The acoustic feature can then be provided to the vocoder 1218 to generate a speech waveform corresponding to the text input.

[0075] It should be understood that Fig.12 The context features 1204 may include references Figure 3 Some or all of the described context features, for example, context feature 1204 may include only current semantic features, historical acoustic features, and historical semantic features. Fig.12 The global features 1210 in may include reference Figure 3 Some or all of the described global features, such as global features 1210 , may include only historical acoustic features and historical semantic features.

[0076] References Figure 2 , Fig.11 and Fig.12 An exemplary neural TTS system according to an embodiment of the present disclosure is described, and accordingly an exemplary method for generating speech through neural TTS synthesis according to an embodiment of the present disclosure is described. The above system and method are intended to generate speech corresponding to text input based on both phoneme features and context features of the text input. It should be understood that the above system and method are exemplary, and the present disclosure is not limited thereto, but other neural TTS systems can be implemented in a similar manner and other methods based on phoneme features and context features can be used to generate speech.

[0077] When generating speech waveforms for a group of sentences, the traditional method usually generates a corresponding speech waveform for each sentence in the group of sentences, and then combines all the generated speech waveforms to obtain a speech waveform corresponding to the group of sentences. A group of sentences is, for example, a paragraph. More specifically, when combining speech waveforms, the pause durations between the speech waveforms corresponding to the sentences are set to be the same or fixed. In order to obtain speech with a richer and more natural rhythm between sentences, the embodiments of the present disclosure further improve the method for generating speech through neural TTS synthesis. Fig.131 is a flow chart of an exemplary method 1300 for generating speech through neural TTS synthesis according to an embodiment of the present disclosure. The method can model inter-sentence pauses and determine the length of inter-sentence pauses according to contextual features.

[0078] At step 1310, text input may be obtained.

[0079] At step 1320, a phoneme sequence may be identified from a text input, such as a current sentence, by various techniques such as LTS.

[0080] At step 1330, the phoneme sequence may be updated by adding a start marker and / or an end marker to the beginning and / or the end of the phoneme sequence, respectively. In one embodiment, a silent phoneme may be used as a start marker and an end marker.

[0081] At step 1340, a phoneme feature may be generated based on the updated phoneme sequence. For example, the updated phoneme sequence may be converted into a phoneme embedding vector sequence by a phoneme embedding model. The phoneme embedding vector sequence may then be provided to an encoder, such as Figure 1 The encoder 110 in , thereby generating phoneme features corresponding to the updated phoneme sequence.

[0082] At step 1350, a speech waveform corresponding to the text input may be generated based on the phoneme features and the context features. The speech waveform may be generated based on a variety of methods.

[0083] In one embodiment, Figure 2 As shown, the phoneme features and the context features can be combined into a mixed feature and the mixed feature can be sequentially provided to the attention unit and the decoder to generate an acoustic feature corresponding to the mixed feature. The acoustic feature can then be provided to the vocoder to generate a speech waveform corresponding to the text input.

[0084] In another embodiment, if Fig.11 As shown, the phoneme feature and the context feature can be combined into a first mixed feature and the first mixed feature can be provided to the attention unit to obtain the first mixed feature with attention implemented. At the same time, the global feature in the context feature can be cascaded with the first mixed feature with attention implemented after passing through its unique attention unit, thereby obtaining a second mixed feature. The second mixed feature can be provided to the decoder to generate an acoustic feature corresponding to the second mixed feature. The acoustic feature can then be provided to the vocoder to generate a speech waveform corresponding to the text input.

[0085] In yet another embodiment, Fig.12As shown, the phoneme feature and the context feature can be combined into a first mixed feature and the first mixed feature can be provided to the attention unit to obtain the first mixed feature with attention implemented. At the same time, the global features in the context feature can be averaged and pooled to obtain an average global feature. The average global feature can be cascaded with the first mixed feature with attention implemented to obtain a second mixed feature. The second mixed feature can be provided to a decoder to generate an acoustic feature corresponding to the second mixed feature. The acoustic feature can then be provided to a vocoder to generate a speech waveform corresponding to the text input.

[0086] Since silent phonemes are added as start and / or end marks at the beginning and / or end of the phoneme sequence of the text input, respectively, the generated speech waveform corresponding to the text input also has a period of silence at the beginning and / or end, respectively. Since context features are taken into account when generating the speech waveform, the silence at the beginning and / or end of the speech waveform is related to the context features, which can vary with different context features. For example, the silence at the end of the speech waveform of the text input and the silence at the beginning of the speech waveform of the next sentence of the text input constitute a pause between the text input and the next sentence. The duration of the pause can also vary with different context features, thereby making the rhythm of the speech corresponding to a group of sentences finally generated richer and more natural.

[0087] Fig.14 An exemplary process 1400 for generating training data for a neural TTS system according to an embodiment of the present disclosure is shown. As described above, the neural TTS system according to an embodiment of the present disclosure can utilize both the phoneme features and the context features of the text input to generate a speech waveform corresponding to the text input. Therefore, the generation of training data for the neural TTS system according to an embodiment of the present disclosure includes phoneme feature generation and context feature generation.

[0088] First, a paragraph text 1402 and a paragraph speech waveform 1404 corresponding to the paragraph text 1402 may be obtained.

[0089] At 1406 , the paragraph text 1402 may be split into a plurality of sentences.

[0090] At 1408, the paragraph speech waveform 1404 may be split into a plurality of speech waveforms corresponding to a plurality of sentences in the paragraph text 1402. During the splitting, the pauses between the sentences may be retained. For example, the pause between two adjacent speech waveforms may be divided into two parts, wherein the first part is attached to the end of the first speech waveform and the second part is attached to the beginning of the second speech waveform.

[0091] The multiple sentences obtained from the paragraph text 1402 and the multiple speech waveforms obtained from the paragraph speech waveform 1404 can correspond one to one to form multiple training data pairs, such as [sentence 1, speech waveform 1], [sentence 2, speech waveform 2], [sentence 3, speech waveform 3], etc. These training data pairs can be used to construct a training corpus.

[0092] A text input, such as a current sentence, may be identified from a training corpus, and a current phoneme sequence 1410 may be identified from the text input. A phoneme feature 1414 may be generated based on the identified current phoneme sequence 1410 by feature generation 1412. For example, reference Figure 1 The phoneme feature 1414 is generated in the manner described for generating the phoneme feature 120 based on the phoneme sequence 104 .

[0093] A current word sequence 1420 may be identified from the text input. A current semantic feature 1424 may be generated based on the identified current word sequence 1420 by feature generation 1422. Figure 4 The described manner of generating the current semantic feature 420 based on the word sequence 404 generates the current semantic feature 1424 .

[0094] A previous segment preceding the text input may be identified from the training corpus, and a previous word sequence 1430 may be identified from the previous segment. A historical semantic feature 1434 may be generated based on the identified previous word sequence 1430 by feature generation 1432. Figure 7 The described manner of generating the historical semantic feature 730 based on the word sequence 704 generates the historical semantic feature 1434 .

[0095] A subsequent segment following the text input may be identified from the training corpus, and a subsequent word sequence 1440 may be identified from the subsequent segment. Future semantic features 1444 may be generated based on the identified subsequent word sequence 1440 by feature generation 1442. Figure 8 The described manner of generating the future semantic feature 830 based on the word sequence 804 generates the future semantic feature 1444 .

[0096] A plurality of sentences in the paragraph in which the text input is located can be identified from the training corpus, and a paragraph word sequence 1450 can be identified from the plurality of sentences. In one embodiment, the paragraph word sequence 1450 can be identified directly from the plurality of sentences in the paragraph. In another embodiment, a central sentence can be first extracted from the plurality of sentences in the paragraph, and then the paragraph word sequence 1450 can be identified from the extracted central sentence. A paragraph semantic feature 1454 can be generated based on the identified paragraph word sequence 1450 through feature generation 1452. For example, reference can be used to generate a paragraph semantic feature 1454. Fig. 9 The paragraph semantic feature 1454 is generated in the manner described for generating the paragraph semantic feature 930 based on the word sequence 904 .

[0097] The position information 1460 of the text input in the paragraph in which it is located can be obtained by training the corpus. The position feature 1464 can be generated based on the extracted position information 1460 by feature generation 1462. For example, reference Fig.10 The location feature 1464 is generated in the same manner as described for generating the location feature 1020 based on the location information 1002 .

[0098] At least one previous acoustic feature 1470 corresponding to at least one previous sentence before the text input can also be obtained by training the corpus. It should be noted that the at least one previous acoustic feature is a real acoustic feature. Taking sentence i as an example of text input, the at least one previous acoustic feature of the text input can be at least one acoustic feature corresponding to at least one speech waveform of at least one sentence before the text input, such as at least one acoustic feature corresponding to speech waveform i-1 of sentence i-1, speech waveform i-2 of sentence i-2, ... speech waveform ik of sentence ik. A historical acoustic feature 1474 can be generated based on the at least one previous acoustic feature 1470 obtained by feature generation 1472. For example, reference can be used to generate a historical acoustic feature 1474. Figure 5 The historical acoustic features 1474 are generated in the manner described for generating the historical acoustic features 520 based on the acoustic features 502 - 1 , 502 - 2 , . . . 502 - k .

[0099] The current semantic feature 1424 , the historical semantic feature 1434 , the future semantic feature 1444 , the paragraph semantic feature 1454 , the position feature 1464 , and the historical acoustic feature 1474 generated in the above manner may be combined into a context feature 1480 .

[0100] according to Fig.14 The phoneme features 1414 and context features 1480 generated by the process shown may be used to train a neural TTS system according to an embodiment of the present disclosure, for example Figure 2 Neural TTS system in 200, Fig.11 Neural TTS system in 1100 or Fig.12 Neural TTS system 1200 in.

[0101] Fig.15 FIG. 1 shows an exemplary process for training a neural TTS system 1500 according to an embodiment of the present disclosure. The neural TTS system 1500 may correspond to Figure 2The neural TTS system 200 in FIG. The neural TTS system 1500 may include at least a cascade unit 1506, an attention unit 1508, a decoder 1510, and a vocoder 1530. The cascade unit 1506, the attention unit 1508, the decoder 1510, and the vocoder 1530 may correspond to Figure 2 The cascade unit 220, attention unit 222, decoder 224 and vocoder 226 in.

[0102] Taking the training of the neural TTS system 1500 using the training data [sentence i, speech waveform i] as an example, the phoneme features 1502 and context features 1504 of sentence i can be obtained first. Fig.14 The phoneme features 1502 and context features 1504 of sentence i are generated in the manner shown.

[0103] The phoneme features 1502 and context features 1504 of sentence i can be combined into mixed features through a cascade unit 1506, and the mixed features can be further processed by an attention unit 1508, a decoder 1510, and a vocoder 1530 to generate a speech waveform i 1540 corresponding to sentence i. The decoder 1510 may include a pre-network 1512, an LSTM 1514, a linear mapping 1516, and a post-network 1518. In the training phase, the input of the pre-network 1512 comes from the real acoustic features 1520 corresponding to the previous speech waveform, i.e., the speech waveform i-1 corresponding to sentence i-1.

[0104] It should be understood that although the foregoing discussion involves Figure 2 The training of the neural TTS system 1500 corresponding to the neural TTS system 200 in FIG. 1 is described in detail below. However, the embodiments of the present disclosure are not limited thereto, but can be trained in a similar manner. Fig.11 Neural TTS system 1100 and Fig.12 Neural TTS system 1200 in.

[0105] Fig.16 is a flow chart of an exemplary method 1600 for generating speech through neural TTS synthesis according to an embodiment of the present disclosure.

[0106] At step 1610, text input may be obtained.

[0107] At step 1620, phonemic features of the text input may be generated.

[0108] At step 1630, a context feature of the text input may be generated based on a set of sentences associated with the text input.

[0109] At step 1640, a speech waveform corresponding to the text input may be generated based on the phoneme features and the context features.

[0110] In one implementation, generating the phoneme feature may include: identifying a phoneme sequence from the text input; and generating the phoneme feature based on the phoneme sequence.

[0111] In one implementation, generating the context feature may include: obtaining an acoustic feature corresponding to at least one sentence in the group of sentences that precedes the text input; and generating the context feature based on the acoustic feature.

[0112] The generating the context feature may further include aligning the context feature with a phoneme sequence of the text input.

[0113] In one implementation, generating the context feature may include: identifying a word sequence from at least one sentence in the group of sentences; and generating the context feature based on the word sequence.

[0114] The at least one sentence may include at least one of: a sentence corresponding to the text input, a sentence located before the text input, and a sentence located after the text input.

[0115] The at least one sentence may represent the content of the group of sentences.

[0116] Generating the contextual feature based on the word sequence may include: generating a word embedding vector sequence based on the word sequence; generating an average embedding vector sequence corresponding to the at least one sentence based on the word embedding vector sequence; aligning the average embedding vector sequence with a phoneme sequence of the text input; and generating the contextual feature based on the aligned average embedding vector sequence.

[0117] In one implementation, generating the context feature may include: determining a position of the text input in the set of sentences; and generating the context feature based on the position.

[0118] Generating the context feature based on the position may include: generating a position embedding vector sequence based on the position; aligning the position embedding vector sequence with a phoneme sequence of the text input; and generating the context feature based on the aligned position embedding vector sequence.

[0119] In one embodiment, generating the speech waveform may include: combining the phoneme feature and the context feature into a mixed feature; implementing an attention mechanism on the mixed feature to obtain an attention-implemented mixed feature; and generating the speech waveform based on the attention-implemented mixed feature.

[0120] In one embodiment, generating the speech waveform may include: combining the phoneme feature and the context feature into a first mixed feature; implementing a first attention mechanism on the first mixed feature to obtain a first mixed feature on which attention is implemented; implementing a second attention mechanism on at least one of the context features to obtain at least one context feature on which attention is implemented; combining the first mixed feature on which attention is implemented and at least one context feature on which attention is implemented into a second mixed feature; and generating the speech waveform based on the second mixed feature.

[0121] In one embodiment, generating the speech waveform may include: combining the phoneme feature and the context feature into a first mixed feature; implementing an attention mechanism on the first mixed feature to obtain a first mixed feature with attention implemented; performing average pooling on at least one of the context features to obtain at least one average context feature; combining the first mixed feature with attention implemented and the at least one average context feature into a second mixed feature; and generating the speech waveform based on the second mixed feature.

[0122] In one embodiment, generating the phoneme feature may include: identifying a phoneme sequence from the text input; updating the phoneme sequence by adding a start tag and / or an end tag to the phoneme sequence, wherein the length of the start tag and the length of the end tag are determined based on the context feature; and generating the phoneme feature based on the updated phoneme sequence.

[0123] In one implementation, the group of sentences may be located in the same paragraph.

[0124] It should be understood that method 1600 may also include any steps / processes for generating speech through neural TTS synthesis according to the above-mentioned embodiments of the present disclosure.

[0125] Fig.17An exemplary device 1700 for generating speech through neural TTS synthesis according to an embodiment of the present disclosure is shown. The device 1700 may include an acquisition module 1710 for obtaining text input; a phoneme feature generation module 1720 for generating phoneme features of the text input; a context feature generation module 1730 for generating context features of the text input based on a set of sentences associated with the text input; and a speech waveform generation module 1740 for generating a speech waveform corresponding to the text input based on the phoneme features and the context features.

[0126] In one implementation, the context feature generation module 1730 may also be configured to: obtain an acoustic feature corresponding to at least one sentence in the group of sentences that precedes the text input; and generate the context feature based on the acoustic feature.

[0127] In one implementation, the context feature generation module 1730 may also be configured to: identify a word sequence from at least one sentence in the group of sentences; and generate the context feature based on the word sequence.

[0128] In one embodiment, the phoneme feature generation module 1720 can also be configured to: identify a phoneme sequence from the text input; update the phoneme sequence by adding a start tag and / or an end tag to the phoneme sequence, wherein the length of the start tag and the length of the end tag are determined based on the context feature; and generate the phoneme feature based on the updated phoneme sequence.

[0129] In addition, the device 1700 may also include any other module configured to generate speech through neural TTS synthesis according to the above-mentioned embodiments of the present disclosure.

[0130] Fig.18 An exemplary apparatus 1800 for generating speech through neural TTS synthesis according to an embodiment of the present disclosure is shown.

[0131] The device 1800 may include at least one processor 1810. The device 1800 may also include a memory 1820 connected to the processor 1810. The memory 1820 may store computer executable instructions, which, when executed, cause the processor 1810 to perform any operation of the method for generating speech through neural TTS synthesis according to the above-mentioned embodiments of the present disclosure.

[0132] The embodiments of the present disclosure may be implemented in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions, which, when executed, cause one or more processors to perform any operation of the method for generating speech through neural TTS synthesis according to the above-mentioned embodiments of the present disclosure.

[0133] It should be understood that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operations in the method or the order of these operations, but should cover all other equivalent transformations under the same or similar concept.

[0134] It should also be understood that all modules in the above described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together functionally.

[0135] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software or any combination thereof. Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraints imposed on the system. As an example, the processor provided in the present disclosure, any part of the processor or any combination of processors can be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gate logic, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The function of the processor provided in the present disclosure, any part of the processor or any combination of processors can be implemented as software executed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0136] Software should be broadly considered to represent instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separated from the processor in the various aspects provided in the present disclosure, the memory can also be located inside the processor (e.g., a cache or a register).

[0137] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described in this disclosure that are known or to be known to those skilled in the art will be covered by the claims.

Claims

1. A method for generating speech by neural text-to-speech (TTS) synthesis, comprising: Get text input; Generating phoneme features of the text input; Generating a context feature of the text input based on a set of sentences associated with the text input, comprising: identifying a word sequence from at least one sentence in the set of sentences; and Generating the context feature based on the word sequence includes: Generate a word embedding vector sequence based on the word sequence; Generating an average embedding vector sequence corresponding to the at least one sentence based on the word embedding vector sequence; aligning the average embedding vector sequence with the phoneme sequence of the text input; and generating the contextual features based on the aligned sequence of average embedding vectors; and A speech waveform corresponding to the text input is generated based on the phoneme features and the context features.

2. The method according to claim 1, wherein: Generating the phoneme feature comprises: identifying a sequence of phonemes from the text input; and The phoneme feature is generated based on the phoneme sequence.

3. The method according to claim 1, wherein: The generating the context feature comprises: Obtaining an acoustic feature corresponding to at least one sentence in the set of sentences that precedes the text input; and The context feature is generated based on the acoustic feature.

4. The method according to claim 3, further comprising: The contextual features are aligned with a phoneme sequence of the text input.

5. The method according to claim 1, wherein: The at least one sentence includes at least one of: a sentence corresponding to the text input, a sentence located before the text input, and a sentence located after the text input.

6. The method according to claim 1, wherein: The at least one sentence represents the content of the group of sentences.

7. The method according to claim 1, wherein: The generating the context feature comprises: determining a position of the text input within the set of sentences; and The context feature is generated based on the location.

8. The method according to claim 7, wherein: The generating the context feature based on the location comprises: generating a sequence of position embedding vectors based on the positions; aligning the position embedding vector sequence with the phoneme sequence of the text input; and The contextual features are generated based on the aligned sequence of position embedding vectors.

9. The method according to claim 1, wherein: The generating of the speech waveform comprises: Combining the phoneme feature and the context feature into a hybrid feature; Implementing an attention mechanism on the mixed feature to obtain an attention-implemented mixed feature; and The speech waveform is generated based on the attention-implemented hybrid feature.

10. The method according to claim 1, wherein: The generating of the speech waveform comprises: Combining the phoneme feature and the context feature into a first mixed feature; Implementing a first attention mechanism on the first mixed feature to obtain a first mixed feature on which attention is implemented; Implementing a second attention mechanism on at least one of the context features to obtain at least one context feature on which attention is implemented; combining the attention-implemented first hybrid feature and the at least one attention-implemented context feature into a second hybrid feature; and The speech waveform is generated based on the second mixed feature.

11. The method according to claim 1, wherein: The generating of the speech waveform comprises: Combining the phoneme feature and the context feature into a first mixed feature; Implementing an attention mechanism on the first mixed feature to obtain a first mixed feature on which attention is implemented; performing average pooling on at least one of the context features to obtain at least one average context feature; combining the attention-implemented first hybrid feature and the at least one average context feature into a second hybrid feature; and The speech waveform is generated based on the second mixed feature.

12. The method according to claim 1, wherein: Generating the phoneme feature comprises: identifying a sequence of phonemes from the text input; updating the phoneme sequence by adding a start tag and / or an end tag to the phoneme sequence, wherein the length of the start tag and the length of the end tag are determined according to the context feature; and The phoneme feature is generated based on the updated phoneme sequence.

13. The method according to claim 1, wherein: The group of sentences are located in the same paragraph.

14. An apparatus for generating speech by neural text-to-speech (TTS) synthesis, comprising: Get module, used to get text input; A phoneme feature generation module, used to generate phoneme features of the text input; A context feature generation module, used to generate context features of the text input based on a group of sentences associated with the text input, comprising: identifying a word sequence from at least one sentence in the set of sentences; and Generating the context feature based on the word sequence includes: Generate a word embedding vector sequence based on the word sequence; Generating an average embedding vector sequence corresponding to the at least one sentence based on the word embedding vector sequence; aligning the average embedding vector sequence with the phoneme sequence of the text input; and generating the contextual features based on the aligned sequence of average embedding vectors; and A speech waveform generation module is used to generate a speech waveform corresponding to the text input based on the phoneme feature and the context feature.

15. The device according to claim 14, wherein: The context feature generation module is further configured to: Obtaining an acoustic feature corresponding to at least one sentence in the set of sentences that precedes the text input; and The context feature is generated based on the acoustic feature.

16. The device according to claim 14, wherein: The phoneme feature generation module is further configured to: identifying a sequence of phonemes from the text input; updating the phoneme sequence by adding a start tag and / or an end tag to the phoneme sequence, wherein the length of the start tag and the length of the end tag are determined according to the context feature; and The phoneme feature is generated based on the updated phoneme sequence.

17. An apparatus for generating speech by neural text-to-speech (TTS) synthesis, comprising: at least one processor; as well as a memory storing computer executable instructions that, when executed, cause the at least one processor to: Get text input; Generating phoneme features of the text input; Generating a context feature of the text input based on a set of sentences associated with the text input, comprising: identifying a word sequence from at least one sentence in the set of sentences; and Generating the context feature based on the word sequence includes: Generate a word embedding vector sequence based on the word sequence; Generating an average embedding vector sequence corresponding to the at least one sentence based on the word embedding vector sequence; aligning the average embedding vector sequence with the phoneme sequence of the text input; and generating the contextual features based on the aligned sequence of average embedding vectors; and A speech waveform corresponding to the text input is generated based on the phoneme features and the context features.

Images