A speech synthesis method and apparatus

Abstract

The present disclosure provides a speech synthesis method and device, wherein the method comprises: determining a context semantic representation corresponding to a target sentence in to-be-processed text data based on a pre-trained cross-sentence text encoder; determining an acoustic representation of a speech waveform corresponding to a historical sentence based on a pre-trained cross-sentence acoustic encoder; wherein the historical sentence is a sentence before the to-be-converted target sentence in the text data; determining a phoneme embedding sequence corresponding to the target sentence, inputting the phoneme embedding sequence into a speech synthesis model to obtain a phoneme hidden state sequence output by the speech synthesis model; and determining a speech waveform corresponding to the target sentence based on the phoneme hidden state sequence, the acoustic representation and the context semantic representation. The speech synthesis method provided by the present disclosure can improve the continuity and richness of the prosody of the synthesized speech, and the generalization ability to unseen fields, thereby improving the naturalness and expressiveness of the synthesized speech.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a speech synthesis method and apparatus. It also relates to an electronic device and a processor-readable storage medium. Background Technology

[0002] In recent years, with the rapid development of deep learning, Text-to-Speech (TTS) systems have achieved near-human-recorded synthesized speech quality in neutral styles. Currently popular end-to-end TTS methods can be divided into two categories: autoregressive models and non-autoregressive models. These neural network models are typically trained using <text, acoustic features> pairs, directly converting the input phoneme sequence or character text sequence into an acoustic feature sequence (such as Mel-domain spectrum) without using any other auxiliary prosodic features. Finally, a neural network vocoder converts the acoustic features into a high-fidelity speech waveform. While current speech synthesis technology can achieve high-fidelity synthesis quality in neutral-style speech, scenarios such as audiobooks, chat rooms, and speeches require higher expressiveness and human-likeness in the synthesized speech, necessitating the synthesis of speech with richer prosody. However, current technologies cannot meet the demands for high prosodic richness and expressiveness. To address this issue, auxiliary features are typically used as additional inputs to TTS. However, due to the need to reference audio input or customize style weights during speech synthesis, large-scale speech synthesis cannot be fully automated. Therefore, designing an efficient speech synthesis scheme to improve the naturalness, expressiveness, and prosodic richness of synthesized speech has become an urgent problem to be solved. Summary of the Invention

[0003] To address these shortcomings, this disclosure provides a speech synthesis method and apparatus to overcome the limitations of existing speech synthesis schemes, which result in poor richness and continuity of synthesized speech prosody and insufficient generalization ability.

[0004] This disclosure provides a speech synthesis method, including:

[0005] Based on a pre-trained cross-sentence text encoder, the contextual semantic representation of the target sentence in the text data to be processed is determined.

[0006] Based on a pre-trained cross-sentence acoustic encoder, the acoustic representation of the speech waveform corresponding to the historical sentence is determined; the historical sentence is the sentence in the text data preceding the target sentence to be converted.

[0007] The phoneme embedding sequence corresponding to the target sentence is determined, and the phoneme embedding sequence is input into the speech synthesis model to obtain the phoneme hidden state sequence output by the speech synthesis model; based on the phoneme hidden state sequence, the acoustic representation, and the contextual semantic representation, the speech waveform corresponding to the target sentence is determined; wherein, the speech synthesis model is trained based on sample text data and the sample speech waveform corresponding to the sample text data.

[0008] Furthermore, the cross-sentence text encoder includes a multi-head attention mechanism layer and a bidirectional semantic representation model;

[0009] The pre-trained cross-sentence text encoder determines the contextual semantic representation of the target sentence in the text data to be processed, specifically including:

[0010] The adjacent sentences in the text data are combined into corresponding sentence pairs. The sentence pairs are then represented at the sentence level based on the bidirectional semantic representation model, and the classification label vector of the sentence pairs is extracted.

[0011] The classification label dimension vector of adjacent sentence pairs in the sentence pair sequence is input into the multi-head attention mechanism layer, and the contextual semantic representation of the target sentence in the text data is output.

[0012] Furthermore, the cross-sentence acoustic encoder includes a long short-term memory network layer and a speech representation model;

[0013] The pre-trained cross-sentence acoustic encoder determines the acoustic representation of the speech waveform corresponding to the historical sentence, specifically including:

[0014] The speech waveform corresponding to the historical sentence preceding the target sentence is determined, and the speech waveform is input into the speech representation model to obtain the comprehensive speech features output by the speech representation model; wherein, the speech representation model is a speech feature extractor obtained through pre-training;

[0015] The integrated speech features are input into the long short-term memory network layer, and the acoustic representation of the speech waveform corresponding to the historical sentence is output.

[0016] Furthermore, determining the phoneme embedding sequence corresponding to the target sentence specifically includes: obtaining the phoneme sequence corresponding to the target sentence;

[0017] The phoneme sequence is input into a preset phoneme embedding layer, and the output information of the phoneme embedding layer is positionally encoded to obtain the phoneme embedding sequence corresponding to the target sentence.

[0018] Furthermore, the speech synthesis model includes a text encoder;

[0019] The phoneme embedding sequence is input into the speech synthesis model to obtain the phoneme hidden state sequence output by the speech synthesis model, specifically including:

[0020] The phoneme embedding sequence is input into the text encoder in the speech synthesis model to obtain the phoneme hidden state sequence output by the text encoder.

[0021] Furthermore, the speech synthesis model also includes a connection layer for merging multiple arrays, a linear layer for feature fusion and adjusting sequence dimensions, a phoneme-level feature predictor, a decoding layer, and a vocoder;

[0022] Based on the phoneme latent state sequence, the acoustic representation, and the contextual semantic representation, the speech waveform corresponding to the target sentence is determined, specifically including:

[0023] The phoneme hidden state sequence output by the text encoder, the acoustic representation, and the contextual semantic representation are concatenated based on the connection layer to obtain the concatenation result.

[0024] Based on the linear layer, feature fusion and sequence dimension adjustment are performed on the splicing result to obtain a cross-sentence hidden state sequence;

[0025] The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information; wherein, the target information includes the phoneme-level fundamental frequency, phoneme-level duration, and phoneme-level energy for representing speech emotion and prosody.

[0026] The target information is position-encoded and then input into the decoding layer to obtain the Mel spectrum of the target sentence output by the decoding layer;

[0027] The vocoder is used to reconstruct the Mel spectrum to obtain the speech waveform corresponding to the target sentence.

[0028] Furthermore, the phoneme-level feature predictor includes a duration warper, a fundamental frequency predictor, an energy predictor, and a duration predictor.

[0029] The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information, specifically including:

[0030] The cross-sentence hidden state sequence is input into the fundamental frequency predictor and the energy predictor in the phoneme-level feature predictor to obtain the phoneme-level fundamental frequency and the phoneme-level energy. The cross-sentence hidden state sequence is input into the duration predictor in the phoneme-level feature predictor to obtain the phoneme-level duration. The phoneme-level sequence is extended to a frame-level sequence by the duration warper to obtain the target information. The phoneme-level sequence includes the phoneme-level fundamental frequency, the phoneme-level energy, and the phoneme-level duration.

[0031] This disclosure also provides a speech synthesis apparatus, including:

[0032] The cross-sentence text representation unit is used to determine the contextual semantic representation of the target sentence in the text data to be processed, based on a pre-trained cross-sentence text encoder.

[0033] A cross-sentence acoustic representation unit is used to determine the acoustic representation of the speech waveform corresponding to a historical sentence based on a pre-trained cross-sentence acoustic encoder; the historical sentence is the sentence in the text data preceding the target sentence to be converted.

[0034] A speech synthesis unit is used to determine the phoneme embedding sequence corresponding to the target sentence, input the phoneme embedding sequence into a speech synthesis model, and obtain the phoneme hidden state sequence output by the speech synthesis model; based on the phoneme hidden state sequence, the acoustic representation, and the contextual semantic representation, the speech waveform corresponding to the target sentence is determined; wherein, the speech synthesis model is trained based on sample text data and the sample speech waveform corresponding to the sample text data.

[0035] Furthermore, the cross-sentence text encoder includes a multi-head attention mechanism layer and a bidirectional semantic representation model;

[0036] The cross-sentence text representation unit is specifically used for:

[0037] The adjacent sentences in the text data are combined into corresponding sentence pairs. The sentence pairs are then represented at the sentence level based on the bidirectional semantic representation model, and the classification label vector of the sentence pairs is extracted.

[0038] The classification label dimension vector of adjacent sentence pairs in the sentence pair sequence is input into the multi-head attention mechanism layer, and the contextual semantic representation of the target sentence in the text data is output.

[0039] Furthermore, the cross-sentence acoustic encoder includes a long short-term memory network layer and a speech representation model;

[0040] The cross-sentence acoustic representation unit is specifically used for:

[0041] The speech waveform corresponding to the historical sentence preceding the target sentence is determined, and the speech waveform is input into the speech representation model to obtain the comprehensive speech features output by the speech representation model; wherein, the speech representation model is a speech feature extractor obtained through pre-training;

[0042] The integrated speech features are input into the long short-term memory network layer, and the acoustic representation of the speech waveform corresponding to the historical sentence is output.

[0043] Furthermore, the speech synthesis unit is specifically used to: obtain the phoneme sequence corresponding to the target sentence;

[0044] The phoneme sequence is input into a preset phoneme embedding layer, and the output information of the phoneme embedding layer is positionally encoded to obtain the phoneme embedding sequence corresponding to the target sentence.

[0045] Furthermore, the speech synthesis model includes a text encoder;

[0046] The speech synthesis unit is specifically used for:

[0047] The phoneme embedding sequence is input into the text encoder in the speech synthesis model to obtain the phoneme hidden state sequence output by the text encoder.

[0048] Furthermore, the speech synthesis model also includes a connection layer for merging multiple arrays, a linear layer for feature fusion and adjusting sequence dimensions, a phoneme-level feature predictor, a decoding layer, and a vocoder;

[0049] The speech synthesis unit is specifically used for:

[0050] The phoneme hidden state sequence output by the text encoder, the acoustic representation, and the contextual semantic representation are concatenated based on the connection layer to obtain the concatenation result.

[0051] Based on the linear layer, feature fusion and sequence dimension adjustment are performed on the splicing result to obtain a cross-sentence hidden state sequence;

[0052] The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information; wherein, the target information includes the phoneme-level fundamental frequency, phoneme-level duration, and phoneme-level energy for representing speech emotion and prosody.

[0053] The target information is position-encoded and then input into the decoding layer to obtain the Mel spectrum of the target sentence output by the decoding layer;

[0054] The vocoder is used to reconstruct the Mel spectrum to obtain the speech waveform corresponding to the target sentence.

[0055] Furthermore, the phoneme-level feature predictor includes a duration warper, a fundamental frequency predictor, an energy predictor, and a duration predictor.

[0056] The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information, specifically including:

[0057] The cross-sentence hidden state sequence is input into the fundamental frequency predictor and the energy predictor in the phoneme-level feature predictor to obtain the phoneme-level fundamental frequency and the phoneme-level energy. The cross-sentence hidden state sequence is input into the duration predictor in the phoneme-level feature predictor to obtain the phoneme-level duration. The phoneme-level sequence is extended to a frame-level sequence by the duration warper to obtain the target information. The phoneme-level sequence includes the phoneme-level fundamental frequency, the phoneme-level energy, and the phoneme-level duration.

[0058] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the speech synthesis method as described in any of the above.

[0059] This disclosure also provides a processor-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described speech synthesis methods.

[0060] The speech synthesis method disclosed herein uses a pre-trained cross-sentence text encoder and a cross-sentence acoustic encoder to extract contextual semantic representations and acoustic representations, and combines these representations with a speech synthesis model. By introducing text-level contextual semantic representations, it overcomes the problem of insufficient semantic information obtained by other works that only use the target sentence and the sentence preceding it. Furthermore, by using the acoustic features extracted by the pre-trained cross-sentence acoustic encoder, it can improve the continuity and richness of the prosodic structure of the synthesized speech, as well as its generalization ability to unseen domains, thereby enhancing the naturalness and expressiveness of the synthesized speech. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic flowchart of the speech synthesis method provided in the embodiments of this disclosure;

[0063] Figure 2 This is a complete flowchart of the speech synthesis method provided in the embodiments of this disclosure;

[0064] Figure 3 This is a schematic diagram of the speech synthesis device provided in the embodiments of this disclosure;

[0065] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0066] To make the technical solutions and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0067] The following describes embodiments based on the speech synthesis method described in this disclosure. Figure 1 As shown, it is a flowchart illustrating the speech synthesis method provided in this embodiment of the present disclosure. The specific implementation process includes the following steps:

[0068] Step 101: Based on the pre-trained cross-sentence text encoder, determine the contextual semantic representation of the target sentence in the text data to be processed.

[0069] The cross-sentence text encoder includes a multi-head attention mechanism layer and multiple bidirectional semantic representation models, such as the BERT (Bidirectional Encoder Representations from Transformer) model. The text data can refer to text data consisting of several consecutive sentences. The contextual semantic representation is the contextual semantic representation data corresponding to the target sentence in the text data to be processed.

[0070] The pre-trained cross-sentence text encoder determines the contextual semantic representation of the target sentence in the text data to be processed. The specific implementation process includes: combining adjacent sentences in the text data into corresponding sentence pair sequences; performing sentence-level representation on the sentence pair sequences based on the bidirectional semantic representation model; extracting the classification label dimension vector of the sentence pair sequences; and then inputting the classification label dimension vectors of adjacent sentence pairs in the sentence pair sequences into a multi-head attention mechanism layer to output the contextual semantic representation of the target sentence in the text data. The bidirectional semantic representation model is a pre-trained Transformer-based bidirectional encoder.

[0071] like Figure 2 As shown, the CU acoustic encoder is a cross-sentence acoustic encoder, LSTM (Long Short-Term Memory) is a long short-term memory network layer, layer summation is a layer summation method layer, and Phoneme sequence of u N-1 The target sentence is represented by the phoneme sequence preceding it; CU text encoder is a cross-sentence text encoder; Multi-head Attention is a multi-head attention mechanism layer; BERT is the BERT (Bidirectional Encoder Representations from Transformer) model; and pair is the sentence pair contained in the text data; Phoneme sequence of u N The target sentence is represented by the phoneme sequence, the encoder is the text encoder, the variation adaptor is the phoneme-level feature predictor, the decoder is the decoder, the concat is the connection layer, the linear projection layer (or linear mapping layer) is the linear layer, the vocoder is the vocoder, the positional encoding is the positional encoder (or positional encoder), the phoneme embedding is the phoneme embedding layer, the Mel-spectrogram is the Mel spectrum, and the waveform is the speech waveform corresponding to the target sentence.

[0072] The text data can be composed of two adjacent sentences forming a corresponding sentence pair sequence, which may include: (u N u N+1 ), (u N+1 u N+2 ), (u N+2 u N+3 )wait.

[0073] Step 102: Based on the pre-trained cross-sentence acoustic encoder, determine the acoustic representation of the speech waveform corresponding to the historical sentence.

[0074] Furthermore, the cross-sentence acoustic encoder includes a long short-term memory network layer (LSTM) and a speech representation model (such as the wav2vec 2.0 model). The historical sentence is the sentence preceding the target sentence in the text data.

[0075] The pre-trained cross-sentence acoustic encoder determines the acoustic representation of the speech waveform corresponding to the historical sentence. The specific implementation process includes: determining the speech waveform corresponding to the historical sentence preceding the target sentence; inputting the speech waveform into the speech representation model to obtain the comprehensive speech features output by the speech representation model; the comprehensive speech features include local acoustic features (i.e., local acoustic information) and text speech features (i.e., text language features). The speech representation model is a pre-trained speech feature extractor; the comprehensive speech features are input into the long short-term memory network layer to output the acoustic representation of the speech waveform corresponding to the historical sentence.

[0076] Furthermore, the specific implementation process for determining the phoneme embedding sequence corresponding to the target sentence includes: obtaining the phoneme sequence of the target sentence. N The phoneme sequence is input into a preset phoneme embedding layer, and the output information of the phoneme embedding layer is positionally encoded to obtain the phoneme embedding sequence corresponding to the target sentence.

[0077] Step 103: Determine the phoneme embedding sequence corresponding to the target sentence, input the phoneme embedding sequence into the speech synthesis model, and obtain the phoneme hidden state sequence output by the speech synthesis model; based on the phoneme hidden state sequence, the acoustic representation, and the contextual semantic representation, determine the speech waveform corresponding to the target sentence; wherein, the speech synthesis model is trained based on sample text data and the sample speech waveform corresponding to the sample text data.

[0078] Furthermore, the speech synthesis model includes a text encoder, a concat layer for merging multiple arrays, a linear projection layer for feature fusion and sequence dimension adjustment, a phoneme-level feature predictor, a decoder, and a vocoder.

[0079] The phoneme embedding sequence is input into the speech synthesis model to obtain the phoneme hidden state sequence output by the speech synthesis model. The specific implementation process is as follows: the phoneme embedding sequence is input into the text encoder in the speech synthesis model for corresponding encoding processing to obtain the phoneme hidden state sequence output by the text encoder.

[0080] Furthermore, based on the phoneme latent state sequence, the acoustic representation, and the contextual semantic representation, the speech waveform corresponding to the target sentence is determined. The specific implementation process includes: concatenating the phoneme latent state sequence, the acoustic representation, and the contextual semantic representation output by the text encoder based on the connection layer to obtain a concatenation result; performing feature fusion and sequence dimension adjustment on the concatenation result based on the linear layer to obtain a cross-sentence latent state sequence; processing the cross-sentence latent state sequence based on the phoneme-level feature predictor to obtain extracted target information; wherein, the target information includes the phoneme-level fundamental frequency, phoneme-level duration, and phoneme-level energy for representing speech emotion and prosody; inputting the target information into the decoding layer after position encoding to obtain the Mel-spectrogram of the target sentence output by the decoding layer; and restoring the Mel-spectrogram based on the vocoder to obtain the speech waveform corresponding to the target sentence.

[0081] Furthermore, the phoneme-level feature predictor includes a duration warper, a fundamental frequency predictor, an energy predictor, and a duration predictor.

[0082] The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information. The specific implementation process includes:

[0083] The cross-sentence hidden state sequence is input into the fundamental frequency predictor and the energy predictor in the phoneme-level feature predictor to obtain the phoneme-level fundamental frequency and phoneme-level energy. The cross-sentence hidden state sequence is also input into the duration predictor in the phoneme-level feature predictor to obtain the phoneme-level duration. The duration warper then expands the phoneme-level sequence to a frame-level sequence to obtain the target information. The phoneme-level sequence includes the phoneme-level fundamental frequency, phoneme-level energy, and phoneme-level duration.

[0084] By using a pre-trained BERT model to extract contextual semantic representations, i.e., cross-utteranc (CU) text, and a pre-trained Wav2Vec 2.0 (i.e., W2V2 model) to extract acoustic representations, and combining these CU representations with speech synthesis models (such as Fastspeech2), the prosodic modeling of speech synthesis and the naturalness and expressiveness of synthesized speech are effectively improved. The CU representations include both CU text and acoustic representations. Specifically, at the textual semantic representation level, document-level cross-utteranc text is introduced through PBE (sentence pair embedding), overcoming the problem of insufficient semantic information obtained by other works that only use the target sentence and the sentence preceding it. Simultaneously, at the acoustic representation level, a W2V2 model pre-trained with massive amounts of cross-lingual speech data is used to extract acoustic features. At the network architecture level, this disclosure adds two additional cross-sentence encoders to the basic speech synthesis model (such as the FastSpeech2 model): one is a cross-sentence acoustic encoder (i.e., the CU acoustic encoder), which uses the W2V2 model to encode acoustic representations of the speech waveforms of the historical sentence (i.e., the sentence preceding the target sentence); the other is a cross-sentence text encoder (i.e., the CU text encoder), which uses the BERT model to encode discourse-level semantic representations of the context text of the target sentence, thus obtaining the contextual semantic representation of the target sentence in the text data. It should be noted that, in order to maintain optimal generalization ability to unseen domains during speech synthesis model training, the parameters of the BERT and W2V2 models in the TTS (Text-to-Speech) framework of this disclosure do not need to be fine-tuned again during speech synthesis model training.

[0085] like Figure 2 As shown, the CU model framework described in this disclosure includes a speech synthesis model, a pre-trained cross-sentence text encoder, and a pre-trained cross-sentence acoustic encoder. Specifically, the CU model framework mainly comprises two modules. The first part is the speech synthesis model, which serves as the basic TTS framework. Its input is a phoneme sequence, and its backbone network is specifically... Figure 2 The middle column; the second part consists of two CU encoders (i.e., a pre-trained cross-sentence text encoder and a pre-trained cross-sentence acoustic encoder), specifically... Figure 2The dashed horizontal box at the midpoint comprises two parts: one is the CU text encoder (i.e., a pre-trained cross-sentence text encoder), which extracts CU text embeddings or CU text, i.e., the corresponding contextual semantic representation, from the pre-trained BERT model; the other is the CU acoustic encoder (i.e., a pre-trained cross-sentence acoustic encoder), which derives CU acoustic embeddings, i.e., acoustic representations, from the historical sentences (preceding sentences) of the target sentence using a W2V2 model. This disclosure uses a multi-head attention mechanism to connect the CU text encoder to the text encoder of the speech synthesis model. In the CU acoustic encoder, an LSTM layer is used to convert unequal-length acoustic embedding sequences into fixed-length sentence-level acoustic representation vectors. Then, the outputs of the two CU encoders are concatenated with the output of the text encoder of the speech synthesis model and further fed to the decoder to generate the Mel-spectrum of the speech. Finally, this disclosure uses a Parallel WaveGAN neural network vocoder to convert the predicted Mel-spectrum into a speech waveform.

[0086] The following is a detailed introduction to each model that makes up the CU model:

[0087] like Figure 2 As shown in the intermediate flow diagram, the speech synthesis model includes at least one text encoder, one phoneme-level feature predictor (Variance adaptor), and one decoder. The text encoder and decoder are multi-layer Transformer structures. The input to the text encoder is a phoneme embedding sequence, and its output is called the phoneme hidden state sequence. Then, the phoneme hidden state sequence is concatenated with the outputs of the two CU encoders, and then passed through a linear layer for feature fusion and sequence dimension adjustment, resulting in the CU hidden state sequence (i.e., the cross-sentence hidden state sequence). The Variance adaptor contains a duration warper and three predictors: a fundamental frequency predictor, an energy predictor, and a duration predictor. In this disclosure, the CU hidden state sequence first predicts the phoneme-level fundamental frequency (i.e., pitch used to express emotion and rhythm) and phoneme-level energy (i.e., energy used to express volume) through a fundamental frequency and energy predictor. Then, it predicts the phoneme-level duration (i.e., phoneme duration). Finally, a duration warper extends the phoneme-level sequence to a frame-level sequence, and the corresponding Mel spectrum is predicted based on the decoder. The phoneme refers to factors that change sound, such as pitch, volume, time, and timbre, ultimately resulting in different spectra generated from the same text data.

[0088] like Figure 2As shown in the dashed box on the right, the CU text encoder is used to extract contextual semantic representations from raw, continuous text data. The CU text encoder incorporates a BERT model and a multi-head attention mechanism. The BERT model is used to extract contextual semantic representations. The BERT model adds a [CLS] dimension before each sentence in the text data as a sentence-level representation. In the CU encoder, adjacent sentences are paired to form sentence pair sequences, and then the classification labels [CLS] of these sentence pair sequences are extracted based on the BERT model. The features composed of these sentence pair [CLS] sequences (i.e., the classification label dimension vector sequence) are called the PBE sequence.

[0089] Suppose a text contains M+L sentences {u N-L ,…,u N ,u N+1 ,…,u N+M}, where the target sentence to be synthesized is u N Therefore, K can be represented as PBEs as,

[0090] E = [e(u N ,u N+1 ),…, u N+K-1 ,u N+K (1)

[0091] Here, e(A, B) represents the [CLS]-dimensional vector obtained by concatenating sentences A and B and feeding them into the BERT model, i.e., the classification label vector. Then, the multi-head attention mechanism calculates the weight coefficients of the value based on the query, key, and value values. In this disclosure, the query and key of this attention mechanism are both PBE matrices E, and the value is the phoneme hidden state sequence of the entire sentence. Finally, the output context vector (i.e., the contextual semantic representation) of the attention mechanism is concatenated with the output of the text encoder in the speech synthesis model (such as the text encoder in Fastspeech2), and further fed into the variance adaptor.

[0092] like Figure 2As shown in the dashed box on the left, the CU acoustic encoder consists of a pre-trained W2V2 model and a single LSTM layer. Its input is the speech waveform of the preceding sentence (i.e., the speech waveform corresponding to the historical sentence). In the W2V2 model, a speech unit represents a speech segment with a frame shift of 20ms and a frame length of approximately 25ms. The W2V2 model's acoustic representation extraction module consists of multiple Transformer layers. It should be noted that the outputs of the earlier Transformer layers in W2V2 contain more local acoustic information, while the outputs of the later layers contain more textual language features. This disclosure extracts the outputs of the earlier, middle, and last Transformer layers of W2V2 and adds them together to obtain a comprehensive speech feature. The aim is to ensure that this representation contains characteristics of different layers in W2V2. Finally, this comprehensive speech feature is input into an LSTM layer to obtain the last hidden state of the LSTM, i.e., the corresponding acoustic representation. This is then concatenated with the output of the text encoder in the speech synthesis model (such as the text encoder in Fastspeech2) and fed into the Variance adaptor.

[0093] The pre-trained BERT and W2V2 models described in this disclosure are used to extract CU text and acoustic representations, which are then used as CU representations and combined with a speech synthesis model. By introducing document-level CU text, this addresses the problem of insufficient semantic information obtained by other works that only use the target sentence and the sentence preceding it. Furthermore, this disclosure utilizes acoustic features extracted by a W2V2 model pre-trained on massive amounts of cross-linguistic speech data. The use of these pre-trained features improves the generalization ability of the CU model to unseen domains, further enhancing the naturalness of the synthesized speech.

[0094] During the training phase, the input to the CU acoustic encoder is the speech waveform corresponding to the historical sentence (i.e., the actual preceding sentence speech), while the input to the CU text encoder is a sequence of sentence pairs. For example... Figure 2As shown, the BERT and W2V2 models, enclosed by dotted dashed lines, are pre-trained with massive amounts of data. To maintain the generalization ability of the pre-trained models (i.e., the BERT and W2V2 models) to unseen domains, the parameters of the BERT and W2V2 models are not updated during the training phase of the TTS model (i.e., the speech synthesis model training phase). It should be noted that for the Chinese dataset, the W2V2 model can be pre-trained using a cross-lingual dataset containing 53 languages; for the English dataset, the W2V2 model is pre-trained using massive amounts of pure English data. This is because there is significantly less open-source data for Chinese compared to English, and using multilingual data for pre-training yields better results than using only a small amount of Chinese data. Furthermore, it should be noted that in this disclosure, the loss function for model training is the same as that for Fastspeech2, including the MSE (mean square error) of fundamental frequency, energy, and duration, and the MAE (mean absolute error) of the Mel-frequency spectrum.

[0095] During the inference phase, the input to the CU acoustic encoder can be either the actual speech corresponding to the preceding text data or synthesized speech (i.e., synthesized preceding audio). There is no significant difference between using the actual preceding speech (i.e., the speech waveform corresponding to the historical sentence) as input and using the synthesized speech input from the speech synthesis model. It should be noted that if synthesized preceding audio is used as input, the process includes the following four steps: 1) Generate preceding audio using any speech synthesis model; 2) Extract the acoustic representation of the synthesized speech using the W2V2 model; 3) Input the acoustic representation of the preceding text (i.e., the historical sentence of the target sentence), and the phoneme sequence of the context semantic text (i.e., the context semantic representation) and the target sentence to synthesize the Mel spectrum of the target sentence; 4) Finally, use a vocoder to restore the Mel spectrum back to the speech waveform corresponding to the target sentence, thus obtaining the desired speech synthesis result.

[0096] The speech synthesis method described in this disclosure uses a pre-trained cross-sentence text encoder and a cross-sentence acoustic encoder to extract contextual semantic representations and acoustic representations, and combines these representations with a speech synthesis model. By introducing text-level contextual semantic representations, it overcomes the problem of insufficient semantic information obtained by other works that only use the target sentence and the sentence preceding it. Furthermore, by using the acoustic features extracted by the pre-trained cross-sentence acoustic encoder, it can improve the continuity and richness of the speech synthesis prosody and the generalization ability to unseen domains, thereby enhancing the naturalness and expressiveness of the synthesized speech.

[0097] Corresponding to the speech synthesis method provided above, this disclosure also provides a speech synthesis apparatus. Since the embodiments of this apparatus are similar to the method embodiments described above, the description is relatively simple. For relevant details, please refer to the description in the method embodiment section above. The embodiments of the speech synthesis apparatus described below are merely illustrative. Please refer to... Figure 3 As shown, it is a structural schematic diagram of a speech synthesis device provided in an embodiment of this disclosure.

[0098] The speech synthesis apparatus described in this disclosure specifically includes the following parts:

[0099] The cross-sentence text representation unit 301 is used to determine the contextual semantic representation of the target sentence in the text data to be processed based on the pre-trained cross-sentence text encoder.

[0100] The cross-sentence acoustic representation unit 302 is used to determine the acoustic representation of the speech waveform corresponding to the historical sentence based on the pre-trained cross-sentence acoustic encoder; the historical sentence is the sentence in the text data preceding the target sentence to be converted.

[0101] The speech synthesis unit 303 is used to determine the phoneme embedding sequence corresponding to the target sentence, input the phoneme embedding sequence into the speech synthesis model, and obtain the phoneme hidden state sequence output by the speech synthesis model; based on the phoneme hidden state sequence, the acoustic representation and the contextual semantic representation, the speech waveform corresponding to the target sentence is determined; wherein, the speech synthesis model is trained based on sample text data and the sample speech waveform corresponding to the sample text data.

[0102] Furthermore, the cross-sentence text encoder includes a multi-head attention mechanism layer and a bidirectional semantic representation model;

[0103] The cross-sentence text representation unit is specifically used for:

[0104] The adjacent sentences in the text data are combined into corresponding sentence pairs. The sentence pairs are then represented at the sentence level based on the bidirectional semantic representation model, and the classification label vector of the sentence pairs is extracted. The bidirectional semantic representation model is a pre-trained bidirectional encoder based on Transformer.

[0105] The classification label dimension vector of adjacent sentence pairs in the sentence pair sequence is input into the multi-head attention mechanism layer, and the contextual semantic representation of the target sentence in the text data is output.

[0106] Furthermore, the cross-sentence acoustic encoder includes a long short-term memory network layer and a speech representation model;

[0107] The cross-sentence acoustic representation unit is specifically used for:

[0108] The speech waveform corresponding to the historical sentence preceding the target sentence is determined, and the speech waveform is input into the speech representation model to obtain the comprehensive speech features output by the speech representation model; the comprehensive speech features include local acoustic information and text language features; wherein, the speech representation model is a speech feature extractor obtained through pre-training;

[0109] The integrated speech features are input into the long short-term memory network layer, and the acoustic representation of the speech waveform corresponding to the historical sentence is output.

[0110] Furthermore, the speech synthesis unit is specifically used to: obtain the phoneme sequence corresponding to the target sentence;

[0111] The phoneme sequence is input into a preset phoneme embedding layer, and the output information of the phoneme embedding layer is positionally encoded to obtain the phoneme embedding sequence corresponding to the target sentence.

[0112] Furthermore, the speech synthesis model includes a text encoder;

[0113] The speech synthesis unit is specifically used for:

[0114] The phoneme embedding sequence is input into the text encoder in the speech synthesis model to obtain the phoneme hidden state sequence output by the text encoder.

[0115] Furthermore, the speech synthesis model also includes a connection layer for merging multiple arrays, a linear layer for feature fusion and adjusting sequence dimensions, a phoneme-level feature predictor, a decoding layer, and a vocoder;

[0116] The speech synthesis unit is specifically used for:

[0117] The phoneme hidden state sequence output by the text encoder, the acoustic representation, and the contextual semantic representation are concatenated based on the connection layer to obtain the concatenation result.

[0118] Based on the linear layer, feature fusion and sequence dimension adjustment are performed on the splicing result to obtain a cross-sentence hidden state sequence;

[0119] The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information; wherein, the target information includes the phoneme-level fundamental frequency, phoneme-level duration, and phoneme-level energy for representing speech emotion and prosody.

[0120] The target information is position-encoded and then input into the decoding layer to obtain the Mel spectrum of the target sentence output by the decoding layer;

[0121] The vocoder is used to reconstruct the Mel spectrum to obtain the speech waveform corresponding to the target sentence.

[0122] Furthermore, the phoneme-level feature predictor includes a duration warper, a fundamental frequency predictor, an energy predictor, and a duration predictor.

[0123] The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information, specifically including:

[0124] The cross-sentence hidden state sequence is input into the fundamental frequency predictor and the energy predictor in the phoneme-level feature predictor to obtain the phoneme-level fundamental frequency and the phoneme-level energy. The cross-sentence hidden state sequence is input into the duration predictor in the phoneme-level feature predictor to obtain the phoneme-level duration. The phoneme-level sequence is extended to a frame-level sequence by the duration warper to obtain the target information. The phoneme-level sequence includes the phoneme-level fundamental frequency, the phoneme-level energy, and the phoneme-level duration.

[0125] The speech synthesis apparatus described in this disclosure uses a pre-trained cross-sentence text encoder and a cross-sentence acoustic encoder to extract contextual semantic representations and acoustic representations, and combines these representations with a speech synthesis model. By introducing text-level contextual semantic representations, it overcomes the problem of insufficient semantic information obtained by other works that only use the target sentence and the sentence preceding it. Furthermore, by using the acoustic features extracted by the pre-trained cross-sentence acoustic encoder, it can improve the continuity and richness of the speech synthesis prosody and the generalization ability to unseen domains, thereby enhancing the naturalness and expressiveness of the synthesized speech.

[0126] Corresponding to the speech synthesis method provided above, this disclosure also provides an electronic device. Since the embodiments of this electronic device are similar to the method embodiments described above, the description is relatively simple. For relevant details, please refer to the description in the method embodiment section above. The electronic device described below is merely illustrative. Figure 4The diagram shows a physical structure of an electronic device disclosed in this embodiment. The electronic device may include a processor 401, a memory 402, and a communication bus 403. The processor 401 and the memory 402 communicate with each other via the communication bus 403 and communicate with external systems via a communication interface 404. The processor 401 can call logical instructions in the memory 402 to execute a speech synthesis method. This method includes: determining the contextual semantic representation of the target sentence in the text data to be processed based on a pre-trained cross-sentence text encoder; determining the acoustic representation of the speech waveform corresponding to a historical sentence based on a pre-trained cross-sentence acoustic encoder; the historical sentence being the sentence in the text data preceding the target sentence to be converted; determining the phoneme embedding sequence corresponding to the target sentence; inputting the phoneme embedding sequence into a speech synthesis model to obtain a phoneme hidden state sequence output by the speech synthesis model; and determining the speech waveform corresponding to the target sentence based on the phoneme hidden state sequence, the acoustic representation, and the contextual semantic representation; wherein the speech synthesis model is trained based on sample text data and sample speech waveforms corresponding to the sample text data.

[0127] Furthermore, the logical instructions in the aforementioned memory 402 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a computer, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as memory chips, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] On the other hand, this disclosure also provides a computer program product, which includes a computer program stored on a processor-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the speech synthesis method provided in the above-described method embodiments. The method includes: determining the contextual semantic representation corresponding to a target sentence in the text data to be processed based on a pre-trained cross-sentence text encoder; determining the acoustic representation of the speech waveform corresponding to a historical sentence based on a pre-trained cross-sentence acoustic encoder; the historical sentence being a sentence in the text data preceding the target sentence to be converted; determining the phoneme embedding sequence corresponding to the target sentence; inputting the phoneme embedding sequence into a speech synthesis model to obtain a phoneme hidden state sequence output by the speech synthesis model; and determining the speech waveform corresponding to the target sentence based on the phoneme hidden state sequence, the acoustic representation, and the contextual semantic representation; wherein the speech synthesis model is trained based on sample text data and sample speech waveforms corresponding to the sample text data.

[0129] In another aspect, embodiments of this disclosure also provide a processor-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis methods provided in the above embodiments. The method includes: determining the contextual semantic representation of a target sentence in text data to be processed based on a pre-trained cross-sentence text encoder; determining the acoustic representation of a speech waveform corresponding to a historical sentence based on a pre-trained cross-sentence acoustic encoder; the historical sentence being a sentence in the text data preceding the target sentence to be converted; determining a phoneme embedding sequence corresponding to the target sentence; inputting the phoneme embedding sequence into a speech synthesis model to obtain a phoneme hidden state sequence output by the speech synthesis model; and determining the speech waveform corresponding to the target sentence based on the phoneme hidden state sequence, the acoustic representation, and the contextual semantic representation; wherein the speech synthesis model is trained based on sample text data and sample speech waveforms corresponding to the sample text data.

[0130] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a computer, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A speech synthesis method, characterized in that, include: Based on a pre-trained cross-sentence text encoder, the contextual semantic representation of the target sentence in the text data to be processed is determined. Based on a pre-trained cross-sentence acoustic encoder, the acoustic representation of the speech waveform corresponding to the historical sentence is determined; the historical sentence is the sentence in the text data preceding the target sentence to be converted. Determine the phoneme embedding sequence corresponding to the target sentence, input the phoneme embedding sequence into the speech synthesis model, and obtain the phoneme hidden state sequence output by the speech synthesis model; Based on the phoneme hidden state sequence, the acoustic representation, and the contextual semantic representation, the speech waveform corresponding to the target sentence is determined; wherein, the speech synthesis model is trained based on sample text data and the sample speech waveforms corresponding to the sample text data; The speech synthesis model includes a text encoder; The phoneme embedding sequence is input into the speech synthesis model to obtain the phoneme hidden state sequence output by the speech synthesis model, specifically including: The phoneme embedding sequence is input into the text encoder in the speech synthesis model to obtain the phoneme hidden state sequence output by the text encoder. The speech synthesis model also includes a connection layer for merging multiple arrays, a linear layer for feature fusion and adjusting sequence dimensions, a phoneme-level feature predictor, a decoding layer, and a vocoder. Based on the phoneme latent state sequence, the acoustic representation, and the contextual semantic representation, the speech waveform corresponding to the target sentence is determined, specifically including: The phoneme hidden state sequence output by the text encoder, the acoustic representation, and the contextual semantic representation are concatenated based on the connection layer to obtain the concatenation result. Based on the linear layer, feature fusion and sequence dimension adjustment are performed on the splicing result to obtain a cross-sentence hidden state sequence; The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information; wherein, the target information includes the phoneme-level fundamental frequency, phoneme-level duration, and phoneme-level energy for representing speech emotion and prosody. The target information is position-encoded and then input into the decoding layer to obtain the Mel spectrum of the target sentence output by the decoding layer; The vocoder is used to reconstruct the Mel spectrum to obtain the speech waveform corresponding to the target sentence.

2. The speech synthesis method according to claim 1, characterized in that, The cross-sentence text encoder includes a multi-head attention mechanism layer and a bidirectional semantic representation model; The pre-trained cross-sentence text encoder determines the contextual semantic representation of the target sentence in the text data to be processed, specifically including: The adjacent sentences in the text data are combined into corresponding sentence pairs. The sentence pairs are then represented at the sentence level based on the bidirectional semantic representation model, and the classification label vector of the sentence pairs is extracted. The classification label dimension vector of adjacent sentence pairs in the sentence pair sequence is input into the multi-head attention mechanism layer, and the contextual semantic representation of the target sentence in the text data is output.

3. The speech synthesis method according to claim 1, characterized in that, The cross-sentence acoustic encoder includes a long short-term memory network layer and a speech representation model; The pre-trained cross-sentence acoustic encoder determines the acoustic representation of the speech waveform corresponding to the historical sentence, specifically including: The speech waveform corresponding to the historical sentence preceding the target sentence is determined, and the speech waveform is input into the speech representation model to obtain the comprehensive speech features output by the speech representation model; wherein, the speech representation model is a speech feature extractor obtained through pre-training; The integrated speech features are input into the long short-term memory network layer, and the acoustic representation of the speech waveform corresponding to the historical sentence is output.

4. The speech synthesis method according to claim 1, characterized in that, The determination of the phoneme embedding sequence corresponding to the target sentence specifically includes: Obtain the phoneme sequence corresponding to the target sentence; The phoneme sequence is input into a preset phoneme embedding layer, and the output information of the phoneme embedding layer is positionally encoded to obtain the phoneme embedding sequence corresponding to the target sentence.

5. The speech synthesis method according to claim 1, characterized in that, The phoneme-level feature predictor includes a duration warper, a fundamental frequency predictor, an energy predictor, and a duration predictor. The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information, specifically including: The cross-sentence hidden state sequence is input into the fundamental frequency predictor and the energy predictor in the phoneme-level feature predictor to obtain the phoneme-level fundamental frequency and the phoneme-level energy. The cross-sentence hidden state sequence is input into the duration predictor in the phoneme-level feature predictor to obtain the phoneme-level duration. The phoneme-level sequence is extended to a frame-level sequence by the duration warper to obtain the target information. The phoneme-level sequence includes the phoneme-level fundamental frequency, the phoneme-level energy, and the phoneme-level duration.

6. A speech synthesis device, characterized in that, include: The cross-sentence text representation unit is used to determine the contextual semantic representation of the target sentence in the text data to be processed, based on a pre-trained cross-sentence text encoder. The cross-sentence acoustic representation unit is used to determine the acoustic representation of the speech waveform corresponding to the historical sentence based on the pre-trained cross-sentence acoustic encoder. The historical sentences are the sentences in the text data that precede the target sentence to be converted. A speech synthesis unit is used to determine the phoneme embedding sequence corresponding to the target sentence, input the phoneme embedding sequence into the speech synthesis model, and obtain the phoneme hidden state sequence output by the speech synthesis model. Based on the phoneme hidden state sequence, the acoustic representation, and the contextual semantic representation, the speech waveform corresponding to the target sentence is determined; wherein, the speech synthesis model is trained based on sample text data and the sample speech waveforms corresponding to the sample text data; The speech synthesis model includes a text encoder; The speech synthesis unit is specifically used for: The phoneme embedding sequence is input into the text encoder in the speech synthesis model to obtain the phoneme hidden state sequence output by the text encoder. The speech synthesis model also includes a connection layer for merging multiple arrays, a linear layer for feature fusion and adjusting sequence dimensions, a phoneme-level feature predictor, a decoding layer, and a vocoder. The speech synthesis unit is specifically used for: The phoneme hidden state sequence output by the text encoder, the acoustic representation, and the contextual semantic representation are concatenated based on the connection layer to obtain the concatenation result. Based on the linear layer, feature fusion and sequence dimension adjustment are performed on the splicing result to obtain a cross-sentence hidden state sequence; The cross-sentence hidden state sequence is processed based on the phoneme-level feature predictor to obtain the extracted target information; wherein, the target information includes the phoneme-level fundamental frequency, phoneme-level duration, and phoneme-level energy for representing speech emotion and prosody. The target information is position-encoded and then input into the decoding layer to obtain the Mel spectrum of the target sentence output by the decoding layer; The vocoder is used to reconstruct the Mel spectrum to obtain the speech waveform corresponding to the target sentence.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the speech synthesis method as described in any one of claims 1 to 5.

8. A processor-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the speech synthesis method as described in any one of claims 1 to 5.

Images