Speech synthesis methods, devices, equipment and storage media

Abstract

This application provides a speech synthesis method, apparatus, device, and storage medium. The specific implementation scheme is as follows: determining the prosodic information of the text to be processed using its semantic information; determining the feature vector of the text to be processed based on its phoneme sequence and the prosodic information; wherein the feature vector of the text to be processed includes: the phoneme sequence of the text to be processed and the prosodic information; synthesizing target speech using the feature vector. According to the technical solution of this application, the expressiveness of the synthesized speech can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, apparatus, device and storage medium. Background Technology

[0002] Text-to-speech (TTS) is a technology that automatically converts text into speech. With the continuous development of deep learning technology and its application in the field of speech synthesis, speech synthesis technology has developed rapidly.

[0003] Current speech synthesis tasks suffer from a "one-to-many" phenomenon, meaning that the same text can correspond to multiple different speech expressions. The minimum mean square error (MSE) criterion is generally used to train the model, but the final model can only learn one averaged prosody, resulting in weak prosodic variation and poor expressiveness in the synthesized speech. Summary of the Invention

[0004] To address the aforementioned problems, this application proposes a speech synthesis method, apparatus, device, and storage medium that can enhance the expressiveness of synthesized speech.

[0005] According to a first aspect of the embodiments of this application, a speech synthesis method is provided, comprising:

[0006] Determine the prosodic information of the text to be processed using its semantic information;

[0007] Based on the phoneme sequence of the text to be processed and the prosodic information, a feature vector of the text to be processed is determined; wherein, the feature vector of the text to be processed includes: the phoneme sequence of the text to be processed and the prosodic information;

[0008] The target speech is synthesized using the feature vectors.

[0009] According to a second aspect of the embodiments of this application, a speech synthesis apparatus is provided, characterized in that it includes:

[0010] The acquisition module is used to determine the prosodic information of the text to be processed using its semantic information;

[0011] The first processing module is used to determine the feature vector of the text to be processed based on the phoneme sequence and the prosodic information; wherein the feature vector of the text to be processed includes: the phoneme sequence and the prosodic information of the text to be processed;

[0012] A synthesis module is used to synthesize target speech using the feature vectors.

[0013] A third aspect of this application provides an electronic device, comprising:

[0014] Memory and processor;

[0015] The memory is connected to the processor and is used to store programs;

[0016] The processor implements the above-described speech synthesis method by running the program in the memory.

[0017] The fourth aspect of this application provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described speech synthesis method.

[0018] One embodiment of the above application has the following advantages or beneficial effects:

[0019] By utilizing the semantic information of the text to be processed, the prosodic information of the text to be processed can be determined, so that the prosodic information can be better matched with the text. Then, based on the phoneme sequence and prosodic information of the text to be processed, a feature vector containing the text, text semantic information and prosodic information can be determined. In this way, the expressiveness of the synthesized speech can be enhanced by using the above feature vectors for speech synthesis. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 A schematic flowchart of a speech synthesis method provided in an embodiment of this application;

[0022] Figure 2 A schematic flowchart of a speech synthesis method S110 provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating the determination of rhythmic gymnastics information provided in an embodiment of this application;

[0024] Figure 4 A schematic diagram illustrating the training of the prosody information extraction model provided in this application embodiment;

[0025] Figure 5 A schematic diagram illustrating the joint training of the first encoder and decoder provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram illustrating a specific process of a speech synthesis method provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of this application are applicable to various speech synthesis scenarios, such as animation scenarios and human-computer interaction scenarios. Using the technical solutions of this application can enhance the expressiveness of synthesized speech.

[0030] The technical solutions of this application can be applied, by way of example, to hardware devices such as processors, electronic devices, and servers (including cloud servers), or packaged as software programs and run. When the hardware device executes the processing procedure of the technical solutions of this application, or when the aforementioned software program is run, the purpose of enhancing the expressiveness of synthesized speech can be achieved. This application only provides illustrative descriptions of the specific processing procedure of the technical solutions of this application and does not limit the specific implementation form of the technical solutions of this application. Any technical implementation form that can execute the processing procedure of the technical solutions of this application can be adopted by this application.

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Exemplary methods

[0033] Figure 1 This is a flowchart of a speech synthesis method according to an embodiment of this application. In an exemplary embodiment, a speech synthesis method is provided, including:

[0034] S110. Determine the prosodic information of the text to be processed using its semantic information;

[0035] S120. Based on the phoneme sequence of the text to be processed and the prosodic information, determine the feature vector of the text to be processed; wherein, the feature vector of the text to be processed includes: the phoneme sequence of the text to be processed and the prosodic information;

[0036] S130. Synthesize the target speech using the feature vector.

[0037] In step S110, exemplarily, the text to be processed represents the text that needs to be converted into speech data. Optionally, the text to be processed can be text directly or indirectly input by the user on a terminal device, or text specified on a webpage or in a text library, without limitation. The terminal device can be a device with communication and input functions, such as a mobile phone, tablet, or handwriting tablet. The semantic information of the text to be processed represents the textual meaning of the text. Optionally, the textual meaning of each sentence can be determined based on the context. Prosodic information is used to represent pauses and variations in the text.

[0038] Specifically, the prosodic information corresponding to the semantic information of the text to be processed can be determined based on the pre-stored mapping relationship between semantic information and prosodic information. The pre-stored mapping relationship between semantic information and prosodic information can be determined based on a large amount of semantic information sample data and prosodic information sample data.

[0039] In step S120, for example, a phoneme is the smallest unit of speech defined based on the natural properties of speech. From an acoustic perspective, a phoneme is the smallest unit of speech defined from the perspective of sound quality. The phoneme sequence of the text to be processed is used to represent the sequence formed by the phonemes corresponding to the text to be processed. Optionally, the encoded vector of the phoneme sequence of the text to be processed and the encoded vector corresponding to the prosodic information are concatenated to obtain a feature vector containing both text and prosodic information.

[0040] In step S130, for example, the feature vector can be decoded by a decoder to obtain the target speech. Specifically, the decoder can be a vocoder, which decodes the feature vector to obtain the final speech waveform.

[0041] In the technical solution of this application, the prosodic information of the text to be processed is determined by using the semantic information of the text to be processed, so that the prosodic information can be better matched with the text. Then, based on the phoneme sequence and prosodic information of the text to be processed, a feature vector containing the text, text semantic information and prosodic information can be determined. In this way, the expressiveness of the synthesized speech can be enhanced by using the above feature vector for speech synthesis.

[0042] In one implementation, such as Figure 2 As shown, the step S110 of determining the prosodic information of the text to be processed using semantic information includes:

[0043] S210. Extract the semantic information of the text to be processed;

[0044] S220. Determine the prosodic information of the text to be processed based on the phoneme sequence and the semantic information.

[0045] For example, semantic information can be extracted from a text semantic encoder, where the text semantic encoder can be a BERT (Bidirectional Encoder Representation from Transformers) model, a pre-trained transformer-based model. The BERT model is pre-trained using text training samples and corresponding semantic information, allowing it to extract semantics from the text to be processed. Semantic information can be fused from the phoneme sequence of the text to be processed to obtain fused information. This fused information is then input into a pre-defined model to obtain the prosodic information of the text. This pre-defined model can be trained using fused training data formed from semantic training data and its corresponding phoneme sequences. Therefore, fusing semantic information with the phoneme sequence of the text to be processed allows for a full understanding of the text's pauses and variations, resulting in a better match between the prosodic information and the text.

[0046] In another implementation, determining the prosodic information of the text to be processed using its semantic information includes:

[0047] The text to be processed is input into a preset prosodic information prediction model to obtain the prosodic information of the text to be processed.

[0048] The prosodic information prediction model includes a text semantic encoder and a prosodic information decoder. The text semantic encoder is used to extract the semantic information of the text to be processed. The prosodic information decoder is used to determine the prosodic information of the text to be processed based on the phoneme sequence of the text to be processed and the semantic information.

[0049] For example, the prosodic information prediction model includes a text semantic encoder and a prosodic information decoder. The text semantic encoder can be a BERT model, and the prosodic information decoder can be a Glow decoder. When training the prosodic information prediction model, the text semantic encoder needs to be trained beforehand to enable it to recognize semantics. Then, based on the trained text semantic encoder, semantics are extracted from the text training samples. The extracted semantic information and the phoneme sequences of the text training samples are then used as training data to train the prosodic information decoder until the model converges, resulting in a trained prosodic information decoder. Thus, a trained prosodic information prediction model is generated through the BERT model and the prosodic information decoder. The additional semantic information provided by the BERT model helps the prosodic information prediction model better model prosodic information such as fundamental frequency fluctuations and stress positions, making the prosody and text more closely matched. Therefore, the trained prosodic information prediction model can directly determine the prosodic information based on the semantic information.

[0050] In one implementation, the step S220 of determining the prosodic information of the text to be processed based on the phoneme sequence and the semantic information includes:

[0051] S310. Encode the phoneme sequence and semantic information of the text to be processed to determine the output sequence; the output sequence includes the phoneme sequence and semantic information of the text to be processed.

[0052] S320. Determine the predicted duration information of each phoneme in the phoneme sequence of the text to be processed based on the output sequence;

[0053] S330. Determine the corresponding first acoustic feature based on the predicted duration information of each phoneme in the output sequence and the phoneme sequence of the text to be processed;

[0054] S340. Determine the prosodic information of the text to be processed using the first acoustic feature.

[0055] Preferably, step S330 includes: using the correspondence between the preset target duration and the predicted duration information of each phoneme in the phoneme sequence of the text to be processed, updating the duration of each output vector in the output sequence to obtain the updated output sequence.

[0056] For example, firstly, the phoneme sequence of the text to be processed is encoded, that is, the discrete phoneme sequence is mapped onto a high-dimensional vector representation. For example, the phoneme sequence is non-linearly encoded using a multi-layer transformer network to obtain the encoded vector of the phoneme sequence. Based on the result of splitting the text into phonemes, the text corresponding to each phoneme is determined, and the semantic information corresponding to the text (i.e., the BERT hidden layer vector) is extracted and concatenated with the encoded vector of the phoneme sequence to obtain the concatenated result. The concatenated result is input into a multi-layer bidirectional long short-term memory (BiLSTM) network for further encoding, and the output vector containing phoneme pronunciation information and semantic information (i.e., the output sequence) is output.

[0057] Secondly, the output sequence is predicted based on the duration model to obtain the predicted duration information for each phoneme. Specifically, the output sequence is re-encoded using a multi-layer BiLSTM network (e.g., three layers) to include more duration-related information in the encoded hidden vectors. Then, a feedforward network directly predicts the duration of each phoneme. It's important to note that the target duration of this duration model (multi-layer BiLSTM network combined with a feedforward network) is predetermined, for example, extracted using a Hidden Markov Model (HMM). During training, the network is updated by backpropagating based on the loss between the target duration and the predicted duration (e.g., the mean squared error loss function). Furthermore, the target duration is input during training, while the predicted duration is input during speech synthesis. In this way, the trained network can predict the predicted duration corresponding to each phoneme, thus effectively synthesizing speech. Finally, the output sequence is upsampled based on the predicted duration information for each phoneme to obtain the first acoustic features. For example, if the output sequence is [h1,h2,h3], the prediction duration is [1,2,2], and the upsampled output is [h1,h2,h2,h3,h3], then the prosodic information of the text to be processed is obtained by decoding the first acoustic feature.

[0058] Further, the step S340 of determining the prosodic information of the text to be processed using the first acoustic feature includes:

[0059] The first acoustic feature is processed using a preset prosodic information conversion model to obtain the prosodic encoding vector; wherein the prosodic information conversion model is obtained by extracting prosodic information from acoustic training samples using the training prosodic encoding vector as training labels, and the training prosodic encoding vector is extracted from speech training data.

[0060] For example, the speech training data can be real speech data collected directly or indirectly with the user's consent, or speech data obtained from an open-source speech library; there is no limitation here. Acoustic training samples represent training data formed based on the acoustic features corresponding to different text training data. The prosodic information conversion model can use the Glow decoder (Generative Flow), or other decoders; there is no limitation here.

[0061] Specifically, when the prosodic information conversion model uses the Glow decoder, the entire Glow decoder consists of multiple blocks, each including an ActNorm layer, an invertible 1*1 convolutional layer, and a coupling layer. Glow's training criterion is maximum likelihood estimation. Let x be the upsampled acoustic features, and y be the target prosodic information (i.e., the training prosodic encoding vector). The training objective function can then be expressed as follows:

[0062]

[0063] Where, p z Let z = f(y|x) be a standard normal distribution, and f be an invertible transformation of y. Since f is an invertible transformation, during synthesis, z only needs to be randomly sampled from the standard normal distribution to generate z, and then the inverse transformation f is applied. -1 Prosodic information is generated by reverse engineering.

[0064] Furthermore, such as Figure 3 As shown, after the Glow decoder is trained, if the text to be processed is obtained, its semantic information is extracted, and the phoneme sequence of the text to be processed is encoded to obtain the encoded vector of the phoneme sequence of the text to be processed. Then, the semantic information and the encoded vector of the phoneme sequence of the text to be processed are concatenated to obtain the output sequence. Based on the output sequence, the predicted duration information of each phoneme in the phoneme sequence of the text to be processed is determined. Based on the predicted duration information of each phoneme in the phoneme sequence of the output sequence and the phoneme sequence of the text to be processed, the corresponding first acoustic feature is determined. The first acoustic feature is input into the Glow decoder to obtain prosodic information. It can be seen that, since it can model prosodic information with arbitrary distribution, the prosodic information conversion model can accurately identify the prosodic information in the text.

[0065] Preferably, the training prosodic encoding vector is extracted from speech training data and includes:

[0066] The speech training data is input into a preset prosodic information extraction model to obtain a training prosodic encoding vector; wherein, the preset prosodic information extraction model is obtained by training a preset neural network model on prosodic information extraction based on positive example training data and negative example training data; the positive example training data and negative example training data are constructed for each target feature vector in the speech database; the positive example training data and negative example training data are obtained by performing inverse causal convolutional encoding on the speech training data in the speech database.

[0067] For example, positive example training data represents speech data segments related to the target feature vector. Negative example training data represents speech data segments unrelated to the target feature vector. It should be noted that the target feature vector can be a randomly selected speech data segment from a speech database, or it can be selected according to specific rules; this is not limited here. The prosodic information extraction model can employ a Long Short-Term Memory network, or other models; this is not limited here.

[0068] In related technologies, there are various ways to extract prosodic codes, most of which adopt predictive learning methods, such as AE (Auto-Encoder) and VAE (Variational Auto-Encoder). However, predictive learning takes the reconstruction of the input as the training objective, which easily extracts too much redundant detailed information.

[0069] Optionally, in the proposed technical solution, causal convolutional coding is performed on the speech training data in the speech library to obtain a first vector; and temporal coding is performed on the first vector to obtain the target feature vector.

[0070] Specifically, such as Figure 4 As shown, the speech training data from the speech database is input into a causal convolutional encoder for complex nonlinear transformation, resulting in the encoded first vector. Then, a long short-term memory network is used to temporally encode this first vector, yielding a target feature vector that serves as the anchor point for the training model. This demonstrates that because the causal convolutional encoder can limit the receptive field of the vector, preventing it from seeing information from future frames, it can prevent cheating during comparative training due to overlap in the receptive fields of the anchor point (i.e., the target feature vector) and the positive training data.

[0071] Furthermore, the speech training data from the speech database is input into a reverse causal convolutional encoder for complex nonlinear transformation, resulting in an encoded second vector. This second vector serves as the positive and negative training data required for the contrastive training criteria. Thus, because the reverse causal convolutional encoder can limit the vector from including information from historical frames, it can prevent cheating during contrastive training. It should be noted that because a large range of positive examples can better extract abstract prosodic information while ignoring detailed information, for each anchor point, a frame is randomly selected within its next 400ms range as the positive training data. The negative training data includes intra-sentence negative examples and global negative examples. Intra-sentence negative examples are selected from all frames within the current sentence and after the positive training data, primarily helping to extract prosodic variation information within the sentence. Global negative examples are selected from any frame of other sentences in the speech database, primarily helping to extract overall sentence style information.

[0072] In this embodiment, the prosody information extraction model employs a contrastive learning training method, focusing on the anchor point c of each target feature vector. t Construct reasonable positive training data and negative example training data set Then, the objective function can be optimized according to the preset training criteria (such as InfoNCE), thus obtaining the preset prosodic information extraction model.

[0073] The objective function formula is as follows:

[0074]

[0075] Where, f(c t , z t )=exp(c′ t ·z t ).

[0076] In one implementation, determining the feature vector of the text to be processed based on the phoneme sequence and the prosodic information includes:

[0077] Based on the phoneme sequence of the text to be processed and the predicted duration information of each phoneme in the phoneme sequence of the text to be processed, the second acoustic features at the frame level of the text to be processed are determined.

[0078] The second acoustic feature is concatenated with the prosodic information to obtain the feature vector of the text to be processed.

[0079] For example, after obtaining the prosodic information and acquiring the predicted duration information of each phoneme in the phoneme sequence of the text to be processed, the encoded vector of the phoneme sequence of the text to be processed and the predicted duration information of each phoneme in the phoneme sequence are upsampled to obtain the frame-level second acoustic feature. Then, the prosodic information and the frame-level second acoustic feature are concatenated, and the concatenation result is encoded. In this way, the prosodic information is added to the frame-level second acoustic feature to obtain a feature vector containing both text and prosodic information, which helps to better determine the speech waveform.

[0080] In one implementation, determining a feature vector of the text to be processed based on the phoneme sequence and the prosodic information, and synthesizing target speech using the feature vector, includes:

[0081] The phoneme sequence and prosodic information of the text to be processed are input into a preset first encoder to obtain the feature vector of the text to be processed;

[0082] The feature vector of the text to be processed is input into a preset decoder, and the target speech is output.

[0083] The decoder is obtained by training speech decoding based on the training feature vector output by the second encoder, which is used to encode the linear spectrum to obtain the training feature vector.

[0084] Preferably, the training method for the decoder and the first encoder includes:

[0085] The first loss function is determined based on the output of the first encoder and the output of the second encoder;

[0086] The decoder and the first encoder are jointly optimized using the first loss function and the second loss function corresponding to the decoding result of the decoder to obtain the trained decoder and the first encoder.

[0087] For example, a pre-defined first encoder is used to predict the prior distribution of the feature vectors. A pre-defined second encoder is used to obtain the posterior distribution of the feature vectors from the linear spectrum. The linear spectrum is obtained by Fourier transforming the expected speech data. It is understood that when training the decoder, the decoder's training data includes the linear spectrum of the speech training data and the speech training data itself, which is obtained from a speech database.

[0088] Specifically, in related technologies, acoustic models and vocoders are typically modeled separately, with acoustic features serving as an intermediate bridge between the two models. This introduces two problems. First, the acoustic features used as a bridge between models are not perfect intermediate features; they are extracted independently of the current synthesis model according to certain rules, and therefore may contain information that the synthesis model does not need, or may lose information that the synthesis model needs. Second, separate modeling leads to a mismatch between training and testing. The vocoder sees the real acoustic features during training, while the synthesis model sees the acoustic features generated by the acoustic model, and there are significant differences between the two. These two reasons greatly impair the synthesized sound quality of the synthesis model.

[0089] In this embodiment, such as Figure 5 As shown, the first encoder includes a phoneme encoder, an upsampling module, a concatenation module, and a flow module. The flow module includes multiple flow layers, such as four layers. The second encoder consists of several convolutional layers. A linear spectrum is input to the second encoder, which outputs a training feature vector; that is, it extracts the bridging features between the first encoder and decoder from the lossless linear spectrum. The decoder can be a vocoder. During training of the first encoder and decoder, the first encoder is trained using the loss between the prior distribution of the feature vector output by the first encoder and the posterior distribution of the feature vector output by the second encoder. For example, the KL divergence loss function, Kullback-Leibler Divergence (i.e., the first loss function), is determined based on the prior distribution of the feature vector output by the first encoder and the posterior distribution of the feature vector output by the second encoder.

[0090] For the decoder, the maximum likelihood loss function (i.e., the second loss function) is used to train the decoder based on the training feature vector output by the second encoder. A target loss function is generated based on the maximum likelihood loss function and the KL divergence loss function, and the decoder is trained using this target loss function. Once the decoder and the first encoder converge, the second encoder is no longer needed. It should be noted that the first and second loss functions can also be other loss functions set according to actual needs; this is not limited here. In this way, joint training allows the decoder and the first encoder to learn a more suitable intermediate feature vector, and avoids the problem of acoustic feature training-test mismatch.

[0091] Specifically, such as Figure 6 As shown, after the first encoder and decoder are trained, if the text to be processed is obtained, the prosodic information corresponding to the text to be processed is first determined. Next, the phoneme sequence of the text to be processed is non-linearly encoded by the phoneme encoder to obtain the phoneme sequence encoding vector. The phoneme sequence encoding vector and the predicted duration information of each phoneme in the phoneme sequence are input into the upsampling module, outputting a frame-level feature vector (i.e., the second acoustic feature at the frame level). Then, the frame-level feature vector and prosodic information are concatenated in the hidden layer dimension by the concatenation module. The concatenation result is encoded by a multi-layer BiLSTM network (such as a two-layer BiLSTM network) to obtain a feature vector containing text and prosodic information. The feature vector containing text and prosodic information is transformed by a four-layer Flow layer to output the feature vector of the text to be processed. The feature vector of the text to be processed is input into the vocoder to output the target speech. In this way, the target speech can be directly output without deciphering the acoustic features, thereby significantly improving the synthesized sound quality of the synthesized speech.

[0092] Exemplary device

[0093] Correspondingly, Figure 7 This is a schematic diagram of a speech synthesis apparatus according to an embodiment of this application. In an exemplary embodiment, a speech synthesis apparatus is provided, comprising:

[0094] The acquisition module 710 is used to determine the prosodic information of the text to be processed using the semantic information of the text to be processed;

[0095] The first processing module 720 is used to determine the feature vector of the text to be processed based on the phoneme sequence and the prosodic information of the text to be processed; wherein, the feature vector of the text to be processed includes: the phoneme sequence and the prosodic information of the text to be processed;

[0096] The synthesis module 730 is used to synthesize target speech using the feature vector.

[0097] In one embodiment, the acquisition module 710 includes:

[0098] An extraction module is used to extract the semantic information of the text to be processed;

[0099] The second processing module is used to determine the prosodic information of the text to be processed based on the phoneme sequence and the semantic information.

[0100] In one embodiment, the second processing module is further configured to:

[0101] The phoneme sequence and semantic information of the text to be processed are encoded to determine the output sequence; the output sequence includes the phoneme sequence and semantic information of the text to be processed.

[0102] The predicted duration information of each phoneme in the phoneme sequence of the text to be processed is determined based on the output sequence.

[0103] Based on the predicted duration information of each phoneme in the output sequence and the phoneme sequence of the text to be processed, the corresponding first acoustic feature is determined;

[0104] The prosodic information of the text to be processed is determined using the first acoustic feature.

[0105] In one implementation, determining the prosodic information of the text to be processed using the first acoustic feature includes:

[0106] The first acoustic feature is processed using a preset prosodic information conversion model to obtain the prosodic encoding vector;

[0107] The prosodic information conversion model is obtained by using the training prosodic encoding vector as the training label to extract prosodic information from the acoustic training samples. The training prosodic encoding vector is extracted from the speech training data.

[0108] In one implementation, the training prosodic encoding vector is extracted from speech training data, including:

[0109] The speech training data is input into a preset prosodic information extraction model to obtain a training prosodic encoding vector;

[0110] The preset prosodic information extraction model is obtained by training a preset neural network model on prosodic information extraction based on positive and negative training data; the positive and negative training data are constructed for each target feature vector in the speech database; the positive and negative training data are obtained by performing inverse causal convolutional encoding on the speech training data in the speech database.

[0111] In one implementation, the method for obtaining the target feature vector includes:

[0112] The speech training data in the speech library is subjected to causal convolutional encoding to obtain a first vector;

[0113] The first vector is temporally encoded to obtain the target feature vector.

[0114] In one implementation, determining the prosodic information of the text to be processed using its semantic information includes...

[0115] The text to be processed is input into a preset prosodic information prediction model to obtain the prosodic information of the text to be processed.

[0116] The prosodic information prediction model includes a text semantic encoder and a prosodic information decoder. The text semantic encoder is used to extract the semantic information of the text to be processed. The prosodic information decoder is used to determine the prosodic information of the text to be processed based on the phoneme sequence of the text to be processed and the semantic information.

[0117] In one embodiment, the first processing module 720 is further configured to:

[0118] Based on the phoneme sequence of the text to be processed and the predicted duration information of each phoneme in the phoneme sequence of the text to be processed, the second acoustic features at the frame level of the text to be processed are determined.

[0119] The second acoustic feature is concatenated with the prosodic information to obtain the feature vector of the text to be processed.

[0120] In one implementation, determining a feature vector of the text to be processed based on the phoneme sequence and the prosodic information, and synthesizing target speech using the feature vector, includes:

[0121] The phoneme sequence and prosodic information of the text to be processed are input into a preset first encoder to obtain the feature vector of the text to be processed;

[0122] The feature vector of the text to be processed is input into a preset decoder, and the target speech is output.

[0123] The decoder is obtained by training speech decoding based on the training feature vector output by the second encoder, which is used to encode the linear spectrum to obtain the training feature vector.

[0124] In one implementation, the training method for the decoder and the first encoder includes:

[0125] The first loss function is determined based on the output of the first encoder and the output of the second encoder;

[0126] The decoder and the first encoder are jointly optimized using the first loss function and the second loss function corresponding to the decoding result of the decoder to obtain the trained decoder and the first encoder.

[0127] The speech synthesis device provided in this embodiment belongs to the same concept as the speech synthesis method provided in the above embodiments of this application. It can execute the speech synthesis method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the speech synthesis method. Technical details not described in detail in this embodiment can be found in the specific processing content of the speech synthesis method provided in the above embodiments of this application, and will not be repeated here.

[0128] Exemplary electronic devices

[0129] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 8 As shown, the device includes:

[0130] Memory 800 and processor 810;

[0131] The memory 800 is connected to the processor 810 and is used to store programs;

[0132] The processor 810 is configured to implement the speech synthesis method disclosed in any of the above embodiments by running the program stored in the memory 800.

[0133] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 820, an input device 830, and an output device 840.

[0134] The processor 810, memory 800, communication interface 820, input device 830, and output device 840 are interconnected via a bus. Among them:

[0135] A bus can include a pathway for transmitting information between various components of a computer system.

[0136] The processor 810 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0137] The processor 810 may include a main processor, as well as a baseband chip, modem, etc.

[0138] The memory 800 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 800 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0139] Input device 830 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0140] Output device 840 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0141] The communication interface 820 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0142] The processor 810 executes the program stored in the memory 800 and calls other devices, which can be used to implement the various steps of any of the speech synthesis methods provided in the above embodiments of this application.

[0143] Exemplary computer program products and storage media

[0144] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the speech synthesis methods according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0145] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0146] Furthermore, embodiments of this application may also be storage media storing computer programs, the computer programs being executed by a processor in the speech synthesis methods according to various embodiments of this application described in the "Exemplary Methods" section above. The specific working content of the above-described electronic device, as well as the specific working content of the computer program product and the computer program on the storage medium being run by a processor, can all be found in the content of the above-described method embodiments, and will not be repeated here.

[0147] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0148] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0149] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0150] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0151] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0152] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0153] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0154] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0156] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0157] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech synthesis method, characterized in that, include: Extract semantic information from the text to be processed; The phoneme sequence and semantic information of the text to be processed are encoded to determine the output sequence; The output sequence includes the phoneme sequence of the text to be processed and the semantic information; The predicted duration information of each phoneme in the phoneme sequence of the text to be processed is determined based on the output sequence. Based on the predicted duration information of each phoneme in the phoneme sequence of the text to be processed, the prosodic information of the text to be processed is determined. Based on the phoneme sequence of the text to be processed and the prosodic information, a feature vector of the text to be processed is determined; wherein, the feature vector of the text to be processed includes: the phoneme sequence of the text to be processed and the prosodic information; The target speech is synthesized using the feature vectors.

2. The method according to claim 1, characterized in that, The step of determining the prosodic information of the text to be processed based on the predicted duration information of each phoneme in the phoneme sequence of the text to be processed includes: Based on the predicted duration information of each phoneme in the output sequence and the phoneme sequence of the text to be processed, the corresponding first acoustic feature is determined; The prosodic information of the text to be processed is determined using the first acoustic feature.

3. The method according to claim 2, characterized in that, The step of determining the prosodic information of the text to be processed using the first acoustic feature includes: The first acoustic feature is processed using a preset prosodic information conversion model to obtain a prosodic encoding vector; The prosodic information conversion model is obtained by using the training prosodic encoding vector as the training label to extract prosodic information from the acoustic training samples. The training prosodic encoding vector is extracted from the speech training data.

4. The method according to claim 3, characterized in that, The training prosodic encoding vector is extracted from speech training data and includes: The speech training data is input into a preset prosodic information extraction model to obtain a training prosodic encoding vector; The preset prosodic information extraction model is obtained by training a preset neural network model on prosodic information extraction based on positive and negative training data; the positive and negative training data are constructed for each target feature vector in the speech database; the positive and negative training data are obtained by performing inverse causal convolutional encoding on the speech training data in the speech database.

5. The method according to claim 4, characterized in that, in, The method for obtaining the target feature vector includes: The speech training data in the speech library is subjected to causal convolutional encoding to obtain a first vector; The first vector is temporally encoded to obtain the target feature vector.

6. The method according to claim 1, characterized in that, The step of determining the feature vector of the text to be processed based on the phoneme sequence and the prosodic information includes: Based on the phoneme sequence of the text to be processed and the predicted duration information of each phoneme in the phoneme sequence of the text to be processed, the second acoustic features at the frame level of the text to be processed are determined. The second acoustic feature is concatenated with the prosodic information to obtain the feature vector of the text to be processed.

7. The method according to claim 1, characterized in that, The feature vector of the text to be processed is determined based on the phoneme sequence and the prosodic information. Synthesizing target speech using the feature vectors includes: The phoneme sequence and prosodic information of the text to be processed are input into a preset first encoder to obtain the feature vector of the text to be processed; The feature vector of the text to be processed is input into a preset decoder, and the target speech is output. The decoder is obtained by training speech decoding based on the training feature vector output by the second encoder, which is used to encode the linear spectrum to obtain the training feature vector.

8. The method according to claim 7, characterized in that, in, The training methods for the decoder and the first encoder include: The first loss function is determined based on the output of the first encoder and the output of the second encoder; The decoder and the first encoder are jointly optimized using the first loss function and the second loss function corresponding to the decoding result of the decoder to obtain the trained decoder and the first encoder.

9. A speech synthesis device, characterized in that, include: The acquisition module is used to extract semantic information from the text to be processed; encode the phoneme sequence and semantic information of the text to be processed, and determine the output sequence; The output sequence includes the phoneme sequence of the text to be processed and the semantic information; Based on the output sequence, determine the predicted duration information of each phoneme in the phoneme sequence of the text to be processed; based on the predicted duration information of each phoneme in the phoneme sequence of the text to be processed, determine the prosodic information of the text to be processed. The first processing module is used to determine the feature vector of the text to be processed based on the phoneme sequence and the prosodic information; wherein the feature vector of the text to be processed includes: the phoneme sequence and the prosodic information of the text to be processed; A synthesis module is used to synthesize target speech using the feature vectors.

10. An electronic device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor implements any one of the speech synthesis methods as described in claims 1 to 8 by running the program in the memory.

11. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements any one of the speech synthesis methods as described in claims 1 to 8.

Images