Method and apparatus for speech synthesis, and device, medium and program product

Abstract

A method and apparatus for speech synthesis, and a device, a medium and a program product. The method comprises: extracting one or more encoded representations corresponding to one or more pieces of target text in a target text set (410); for each encoded representation of the one or more encoded representations, generating, on the basis of each encoded representation and reference speech, a dedicated training sequence corresponding to each encoded representation (420); and training a speech synthesis model on the basis of the dedicated training sequences respectively corresponding to the one or more encoded representations, wherein the speech synthesis model is configured to convert input text into a corresponding speech signal (430).

Description

Methods, apparatus, devices, media, and programs for speech synthesis

[0001] This application claims priority to Chinese Patent Application No. 202411466076.3, filed on October 18, 2024, entitled "Method, Apparatus, Device and Readable Medium for Speech Synthesis", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and more specifically, to methods, apparatus, devices, computer-readable storage media, and computer program products for speech synthesis. Background Technology

[0003] With the advancement of machine learning technology, speech synthesis technology has also developed rapidly. Speech synthesis is a technology that converts arbitrary text information into speech information based on a text-to-speech (TTS) model. TTS models can acoustically model continuous acoustic features and determine the acoustic features corresponding to the text information based on the established acoustic model to output sound information. However, current speech synthesis technology still has some problems, which affect the quality of the output speech information and reduce the user experience. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for speech synthesis is provided. The method includes extracting one or more coded representations corresponding to one or more target texts in a target text set; generating a dedicated training sequence for each coded representation based on each coded representation and a reference speech, for each coded representation; and training a speech synthesis model based on the dedicated training sequences corresponding to each of the one or more coded representations, the speech synthesis model being configured to convert input text into a corresponding speech signal.

[0005] In a second aspect of this disclosure, an apparatus for speech synthesis is provided. The apparatus includes an extraction module configured to extract one or more coded representations corresponding to one or more target texts in a target text set; a generation module configured to generate a dedicated training sequence for each of the one or more coded representations, based on each coded representation and a reference speech; and a training module configured to train a speech synthesis model based on the dedicated training sequences corresponding to each of the one or more coded representations, the speech synthesis model being configured to convert input text into a corresponding speech signal.

[0006] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the methods of the first aspect of this disclosure when executed by the at least one processor.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to perform the method according to a first aspect of this disclosure.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0009] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] In the following detailed description, in conjunction with the accompanying drawings, the above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 shows an example architecture diagram of a speech synthesis model according to some embodiments of the present disclosure;

[0013] Figure 3 illustrates an example architecture diagram of a predictor model according to some embodiments of the present disclosure;

[0014] Figure 4 shows a flowchart of a speech synthesis process according to some embodiments of the present disclosure;

[0015] Figure 5 shows a block diagram of an apparatus for speech synthesis according to some embodiments of the present disclosure; and

[0016] Figure 6 shows a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multi-layered neural networks to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," "machine learning network," "neural network," or simply "network," and these terms are used interchangeably.

[0020] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0021] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0022] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0024] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0025] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0026] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in Figure 1, model 130-1 with pre-training parameter values ​​and model 130-2 with post-training parameter values ​​can be collectively or individually referred to as model 130. Model 130 may be implemented or included in electronic device 140 and / or electronic device 150.

[0027] In environment 100 of Figure 1, it is desirable to train and use a machine learning model (i.e., model 130) configured for various application environments. For example, in the case of a speech synthesis model, speech corresponding to speech can be generated based on target text input by the user.

[0028] As shown in Figure 1, environment 100 includes electronic device 140 and electronic device 150. Electronic device 140 may contain a speech synthesis system, and electronic device 150 may contain a model application system. The upper part of Figure 1 shows the model training phase, and the lower part shows the model application phase. Before training, the parameter values ​​of model 130 can have initial values, or they can have pre-trained parameter values ​​obtained through a pre-training process. Model 130-1 can be trained via forward and backward propagation, during which the parameter values ​​of model 130-1 can be updated and adjusted. After training, model 130-2 is obtained. At this point, the parameter values ​​of model 130-2 have been updated. Based on the updated parameter values, model 130-2 can be used to implement the speech synthesis task in the model application phase.

[0029] During the model training phase, model 130 can be trained using a model training system based on a training sample set 110 comprising multiple training samples 112. Each training sample 112 can involve a binary format. For example, for a speech synthesis task, training sample 112 can include training input 120 and training output for the speech synthesis task. The training input for the speech synthesis task can, for example, include training text and corresponding audio. Training sample 112, including model input 120 and model output 122, can be used to train model 130. Specifically, the training process can be performed iteratively using a large number of training samples. After training is complete, model 130 can include knowledge about the task to be processed. During the model application phase, model 130 (at this point, model 130 has the trained parameter values) can be used to perform the corresponding task. For example, model input 142 for the speech synthesis task can be received, and a corresponding model output 144 can be output.

[0030] In some embodiments, the model training system may also consist of a construction part (e.g., a sample construction subsystem) for constructing training samples and a training part (e.g., a training subsystem) for training the model, thereby separating the sample construction and model training stages. For example, the model training system may consist of a set of devices, with one part serving as the construction part and another part serving as the training part. In some embodiments, the model training system may also utilize the same devices to construct training samples and train the model based on the training samples. This disclosure is not intended to be limiting.

[0031] For ease of understanding, the following examples will all use the same equipment to build training samples and train the model based on the training samples.

[0032] In Figure 1, electronic devices 140 and 150 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. Terminal devices may involve any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Servers include, but are not limited to, mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0033] It should be understood that the components and arrangements in environment 100 shown in Figure 1 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. Implementations of this disclosure are not limited in this respect.

[0034] As briefly mentioned earlier, speech synthesis models can synthesize speech based on user-input text information. Currently, most end-to-end speech synthesis is performed using speech synthesis models based on Transformer network architecture and speech synthesis models based on Generative Adversarial Networks (GANs). To improve model performance, the speech synthesis model needs to be trained using multiple pairs of text and audio training samples to update model parameters and establish a mapping from text to audio features.

[0035] In training a TTS model, the first step is to determine the training samples used for training. These training samples include training text-audio pairs from multiple speakers, where the training audio consists of the speech delivered by each speaker to the training text. However, due to differences in the context of the training text and the speaker's state, even training audio from the same speaker to the same training text may exhibit variations in pronunciation, rhythm, and delivery. Therefore, the obtained training samples may contain "one-to-many" pairs, meaning different training audios correspond to the same training text.

[0036] Typically, the performance of synthesized speech by a speech synthesis model is based on the average performance of the data. That is, the sound quality and prosody learned and synthesized by the model are an average of all training data. If there is a "one-to-many" situation in the training samples, the model trained based on the average performance of the training samples may have bland prosodic performance, which will affect the performance of the speech synthesis model.

[0037] Embodiments of this disclosure provide a method for speech synthesis. According to embodiments of this disclosure, one or more coded representations corresponding to one or more target texts in a target text set are extracted. For each of the one or more coded representations, a dedicated training sequence corresponding to each coded representation is generated based on each coded representation and a reference speech. Based on the dedicated training sequences corresponding to each of the one or more coded representations, a speech synthesis model is trained, the speech synthesis model being configured to convert input text into a corresponding speech signal.

[0038] In this manner, embodiments of this disclosure generate a dedicated training sequence for each encoded representation based on each encoded representation and a reference speech, to train the speech synthesis model. This prevents multiple speech samples from corresponding to the same text, increasing the diversity of training samples. Furthermore, it improves the performance of the speech synthesis model.

[0039] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0040] Figure 2 illustrates an example architecture diagram of a speech synthesis model 200 according to some embodiments of the present disclosure. For ease of discussion, the architecture 200 will be described with reference to the environment 100 of Figure 1.

[0041] In some embodiments, the electronic device 140 uses encoder model 220 to extract one or more encoded representations 221 corresponding to one or more target texts 210 in the target text set.

[0042] In some embodiments, the target text set includes multiple target texts 210 used for training a speech synthesis model. The language type and text length of the target texts 210 are not limited. The electronic device 140 acquires the phoneme sequence 211 corresponding to each target text 210. The phoneme sequence 211 refers to the order of phonemes in a word or phrase in phonetics. The electronic device 140 can determine the phoneme sequence 211 corresponding to the target text 210 based on a dictionary or database. Alternatively or additionally, the electronic device 140 can determine the phoneme sequence 211 corresponding to the target text 210 using Natural Language Processing (NLP) technology or specialized speech analysis software.

[0043] Electronic device 140 acquires a set of phoneme sequences related to a target text set, wherein phoneme sequences 211 in the phoneme sequence set correspond to target text 210 in the target text set. Subsequently, electronic device 140 uses encoder model 220 to perform encoding operations on one or more target phoneme sequences 211 in the phoneme sequence set to generate one or more encoded representations 221.

[0044] In some embodiments, a phoneme sequence 211 corresponding to the target text 210 is provided to an encoder model 220. The electronic device 140 uses the encoder model 220 to perform positional encoding on the provided phoneme sequence 211 to obtain an encoded representation 221 (i.e., a phoneme embedding sequence) corresponding to the target text 210.

[0045] In some embodiments, human speech in a natural paragraph is continuous. The speech corresponding to the target text 210 may be influenced by factors such as relative sentence position, discourse relationships in adjacent sentences, and the sentiment of the surrounding sentences. The electronic device 140 can incorporate the context corresponding to the target text 210 into the model.

[0046] For each of the one or more coded representations 221, the electronic device 140 generates a dedicated training sequence 291 corresponding to each coded representation 221 based on each coded representation 221 and the reference speech 281.

[0047] In some embodiments, when the encoded representation 221 of the target text 210 is determined, the electronic device 140 determines the reference speech 281 corresponding to the target text 210 based on the correspondence between training text and training speech. For example, if the target text 210 and the reference audio exist as data pairs in the training samples, the reference speech 281 corresponding to the target text 210 can be determined based on the target text 210-reference audio pair. In some embodiments, the reference speech 281 can be generated based on the target text 210. Based on the determined reference speech 281 and the encoded representation 221, a dedicated training sequence 291 corresponding to each encoded representation 221 is generated. For example, a speech library including the speech of multiple speakers can be pre-established, and training data including multiple target text 210-speaker identification 280 (Speaker ID) pairs can be generated to reduce the storage space occupied by the training data. During training, the reference speech 281 corresponding to the target text 210 is determined based on the speaker identification 280.

[0048] In some embodiments, the electronic device 140 can generate a dedicated training sequence 291 by setting a unique identifier. For any encoded representation 221 and its reference speech 281, the electronic device 140 obtains identification information 290 for identifying the reference speech 281. Based on the identification information 290 of the reference speech 281, the target encoded representation 221, and the reference speech 281, the dedicated training sequence 291 is generated. The identification information 290 for representing the reference speech 281 can be random data (e.g., random noise signal, etc.) or predetermined information corresponding to the reference speech 281 (e.g., an identity identifier 280 generated based on the length, time, etc. of the reference speech 281, etc.). For example, the noise signal can be a Gaussian noise signal.

[0049] After acquiring the identification information 290, the electronic device 140 generates a dedicated training sequence 291 for the reference speech 281 based on the identification information 290, the reference speech 281, and the encoded representation 221. In some embodiments, the dedicated training sequence 291 can be generated directly by performing a concatenation operation on the target encoded representation 221, the training noise signal, and the reference speech 281. For example, firstly, the speaker embedding 282 corresponding to the reference speech 281 is determined, and then the identification information 290 corresponding to the reference speech 281 is determined. A concatenation operation is performed on the training noise signal and the speaker embedding 282 to obtain a first concatenation result. The first concatenation result is processed using a linear layer to obtain a second concatenation result with the same dimension as the target encoded representation 221. The second concatenation result and the target encoded representation 221 are fused to generate the dedicated training sequence 291. The target text 210 in each dedicated training sequence 291 can be the same text, but with different reference speeches 281 and unique identification information 290 corresponding to the reference speech 281. Therefore, each dedicated training sequence 291 is different. In this way, the diversity of training data can be improved, and the "one-to-many" situation can be prevented in the training samples.

[0050] Alternatively or additionally, operations can be performed on the encoded representation 221 and the reference speech 281 corresponding to the encoded representation 221 using a predetermined processing method (e.g., encryption processing) to determine a unique dedicated training sequence 291 corresponding to the reference speech 281.

[0051] In some embodiments, the electronic device 140 trains a speech synthesis model based on dedicated training sequences 291 corresponding to one or more coded representations 221.

[0052] For a target text set including multiple target texts 210, the electronic device 140, after determining a dedicated training sequence 291 corresponding to each target text 210, trains a speech synthesis model based on the multiple dedicated training sequences 291 corresponding to each target text 210. The speech synthesis model is configured to convert input text into corresponding speech signals.

[0053] In some embodiments, the electronic device 140 first performs phoneme prediction on a dedicated training sequence 291 using a predictor model 230 to generate a predicted phoneme sequence. Then, based on the predicted phoneme sequence, a decoder model 240 generates predicted acoustic features 250. The electronic device 140 trains a speech synthesis model based on the predicted acoustic features 250.

[0054] For any target text 210 in the target text set, the electronic device 140 first determines the predicted acoustic feature 250 corresponding to the target text 210. The predicted acoustic feature 250 can be a Mel-spectral acoustic feature, discrete markers, etc. Subsequently, the electronic device 140 determines the reference acoustic feature 260 corresponding to the target text 210. The reference acoustic feature 260 can be a feature generated based on the reference speech 281. The type of the reference acoustic feature 260 is the same as the type of the predicted acoustic feature 250. Based on the difference between the reference acoustic feature 260 and the predicted acoustic feature 250, the electronic device 140 determines the training loss 271. Subsequently, the electronic device 140 updates the parameters of the speech synthesis model based on the training loss 271.

[0055] In some embodiments, the updated phoneme embedding sequence (i.e., the phoneme embedding sequence incorporating semantic features) is provided to predictor model 230. Predictor model 230 performs phoneme prediction based on the phoneme embedding sequence to obtain a predicted phoneme sequence. Electronic device 140 determines training loss 271 based on loss function 270 and the difference between reference acoustic features 260 and predicted acoustic features 250.

[0056] Figure 3 illustrates an example architecture diagram of a predictor model 300 according to some embodiments of the present disclosure. As shown in Figure 3, the predictor model 230 includes at least an energy prediction unit 310, a fundamental frequency prediction unit 320, a duration prediction unit 330, and a length adjuster 340.

[0057] The duration prediction unit 330 is configured to determine predicted duration information representing the duration of a phoneme. The fundamental frequency prediction unit 320 is configured to determine predicted fundamental frequency information representing speech emotion and prosody. The energy prediction unit 310 is configured to determine predicted energy information representing volume information. As shown in Figure 3, the energy prediction unit 310, the fundamental frequency prediction unit 320, and the duration prediction unit 330 have similar network structures, each including the linear layer, normalization layer, dropout layer, and convolutional layer (Conv1D+ReLU) shown in Figure 3. The length adjuster is configured to upsample the predicted phoneme sequence according to the phoneme duration (i.e., the number of Mel spectrograms corresponding to each phoneme) to match the length of the Mel spectrogram sequence.

[0058] In some embodiments, performing phoneme prediction on the phoneme embedding sequence corresponding to the target text 210 using the predictor model 230 includes: performing phoneme prediction on the provided phoneme embedding sequence using the predictor model 230 to obtain predicted fundamental frequency information for representing speech emotion and prosody, predicted duration information for representing phoneme duration, and predicted energy information for representing volume information. Based on the predicted fundamental frequency information, predicted duration information, and predicted energy information, a frame-level predicted phoneme sequence is determined.

[0059] In some embodiments, the predictor model 230 may be a recurrent neural network (RNN), a long short-term memory network (LSTM), a gated recurrent unit (GRU), or a Transformer-based prediction model.

[0060] In some embodiments, since the length of the speech waveform is much greater than the length of the phoneme sequence 211, to better generate the speech waveform from the predicted phoneme sequence, the phoneme-level predicted phoneme sequence is broadcast into a frame-level predicted phoneme sequence. Specifically, firstly, the number of repetitions of each phoneme is determined based on the unit duration of each frame of the reference audio and the prediction duration information of each phoneme in the predicted phoneme sequence. Subsequently, each phoneme in the predicted phoneme sequence is processed according to the obtained number of repetitions of each phoneme to obtain a frame-level predicted phoneme sequence. For example, if the unit duration of each frame of the reference audio is three times the prediction time of each factor, each phoneme is copied three times to obtain an expanded predicted phoneme sequence.

[0061] In some embodiments, after the speech synthesis model has been trained, it can be used to process the inference text to generate a speech signal related to the inference text. For example, electronic device 150 first acquires the inference text as input to the speech synthesis model. A noise signal related to the inference text is generated. Subsequently, based on the noise signal and the inference text, electronic device 140150 uses the trained speech synthesis model to generate a speech signal corresponding to the inference text. The noise signal can be a randomly generated Gaussian noise signal. In this way, by adding a randomly generated Gaussian noise signal, the prosodic richness and diversity of the synthesized speech signal are increased. Furthermore, the stability of the synthesis effect is improved.

[0062] In some embodiments, a speaker can be specified for the inference text. The speech signal corresponding to the inference text is determined based on the specified speaker.

[0063] In summary, by splicing random noise signals into the speaker embedding, the "one-to-many" problem that may be encountered during model parameter iteration during model training is solved, stimulating the model's parameter optimization ability and improving the training effect. This further improves the model's performance, effectively enhancing the sound quality, prosodic expressiveness, richness, and diversity of the synthesized audio. During the inference phase, the introduction of random Gaussian noise further enhances the sound quality, prosodic expressiveness, richness, and diversity of the synthesized audio.

[0064] Figure 4 shows a flowchart of a speech synthesis process 400 according to some embodiments of the present disclosure. Process 400 can be implemented at an electronic device 140.

[0065] In box 410, extract one or more encoded representations corresponding to one or more target texts in the target text set.

[0066] In some embodiments, determining one or more encoded representations corresponding to one or more target texts in a target text set includes: obtaining a set of phoneme sequences related to the target text set, wherein the phoneme sequences in the phoneme sequence set correspond to the target texts in the target text set; and performing encoding operations on one or more target phoneme sequences in the phoneme sequence set to generate one or more encoded representations.

[0067] In box 420, for each of the one or more coding representations, a dedicated training sequence is generated based on each coding representation and a reference speech.

[0068] In some embodiments, generating a dedicated training sequence related to the target coded representation based on the target coded representation and the reference speech includes: obtaining identification information for identifying the reference speech; and generating the dedicated training sequence based on the identification information of the reference speech, the target coded representation, and the reference speech.

[0069] In some embodiments, the identification information is a training noise signal for the reference speech, and generating a dedicated training sequence includes: generating a dedicated training sequence by performing a concatenation operation on the target encoded representation, the training noise signal, and the reference speech.

[0070] In box 430, a speech synthesis model is trained based on one or more dedicated training sequences corresponding to their respective encoded representations. The speech synthesis model is configured to convert input text into corresponding speech signals.

[0071] In some embodiments, training a speech synthesis model based on one or more dedicated training sequences corresponding to each of the dedicated training sequences includes: performing phoneme prediction on the dedicated training sequences to generate a predicted phoneme sequence; generating predicted acoustic features based on the predicted phoneme sequence; and training the speech synthesis model based on the predicted acoustic features.

[0072] In some embodiments, training a speech synthesis model based on predicted acoustic features includes: determining a given target text in a target text set that corresponds to the predicted acoustic features; determining a reference acoustic feature corresponding to the given target text based on the feature types of the given target text and the predicted acoustic features; determining a training loss based on the difference between the reference acoustic feature and the predicted acoustic feature; and updating the parameters of the speech synthesis model based on the training loss.

[0073] In some embodiments, performing phoneme prediction on a dedicated training sequence to generate a predicted phoneme sequence includes: performing phoneme prediction on the dedicated training sequence to obtain predicted fundamental frequency information for representing speech emotion and prosody, predicted duration information for representing phoneme duration, and predicted energy information for representing volume information; and determining the predicted phoneme sequence based on the predicted fundamental frequency information, the predicted duration information, and the predicted energy information.

[0074] In some embodiments, process 400 further includes: in response to the completion of training of the speech synthesis model, obtaining inference text for input to the speech synthesis model; generating a noise signal related to the inference text; and generating a speech signal corresponding to the inference text using the trained speech synthesis model based on the noise signal and the inference text.

[0075] Figure 5 shows a block diagram of an apparatus 500 for model training according to some embodiments of the present disclosure. The apparatus 500 may be implemented as or included in an electronic device 140.

[0076] Apparatus 500 includes an extraction module 510 configured to extract one or more coded representations corresponding to one or more target texts in a target text set. Apparatus 500 also includes a generation module 520 configured to generate a dedicated training sequence for each of the one or more coded representations, based on each coded representation and a reference speech. Apparatus 500 further includes a training module 530 configured to train a speech synthesis model based on the dedicated training sequences corresponding to each of the one or more coded representations, the speech synthesis model being configured to convert input text into a corresponding speech signal.

[0077] In some embodiments, the extraction module 510 is further configured to acquire a set of phoneme sequences related to a target text set, wherein the phoneme sequences in the phoneme sequence set correspond to target text in the target text set; and to perform encoding operations on one or more target phoneme sequences in the phoneme sequence set to generate one or more encoded representations.

[0078] In some embodiments, the generation module 520 is further configured to acquire identification information for identifying reference speech; and to generate a dedicated training sequence based on the identification information of the reference speech, the target coded representation, and the reference speech.

[0079] In some embodiments, the generation module 520 is further configured to generate a dedicated training sequence by performing a splicing operation on the target coded representation, the training noise signal, and the reference speech.

[0080] In some embodiments, the training module 530 is further configured to perform phoneme prediction on a dedicated training sequence to generate a predicted phoneme sequence; generate predicted acoustic features based on the predicted phoneme sequence; and train a speech synthesis model based on the predicted acoustic features.

[0081] In some embodiments, the training module 530 is further configured to: determine a given target text in the target text set corresponding to the predicted acoustic features; determine a reference acoustic feature corresponding to the given target text based on the feature types of the given target text and the predicted acoustic features; determine a training loss based on the difference between the reference acoustic features and the predicted acoustic features; and update the parameters of the speech synthesis model based on the training loss.

[0082] In some embodiments, the training module 530 is further configured to perform phoneme prediction on a dedicated training sequence to obtain predicted fundamental frequency information for representing speech emotion and prosody, predicted duration information for representing phoneme duration, and predicted energy information for representing volume information; and to determine a predicted phoneme sequence based on the predicted fundamental frequency information, predicted duration information, and predicted energy information.

[0083] In some embodiments, the apparatus 500 further includes: an inference module configured to, in response to the completion of training of the speech synthesis model, acquire inference text for input to the speech synthesis model; generate a noise signal related to the inference text; and, based on the noise signal and the inference text, generate a speech signal corresponding to the inference text using the trained speech synthesis model.

[0084] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0085] Figure 6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in Figure 6 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 600 may be used to implement the apparatus 500 of Figure 5 or the electronic device 140 of Figure 1.

[0086] As shown in Figure 6, the electronic device 600 is in the form of a general-purpose computing device. Components of the electronic device 600 may include, but are not limited to, one or more processors or processor 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processor 610 may be a physical or virtual processor and is capable of performing various processes based on programs stored in memory 620. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

[0087] Electronic device 600 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 600.

[0088] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 6, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0089] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0090] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0091] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transient computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above.

[0092] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0094] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0096] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A speech synthesis method, comprising: Extract one or more encoded representations corresponding to one or more target texts in the target text set; For each of the one or more coding representations, a dedicated training sequence is generated based on each coding representation and a reference speech. as well as A speech synthesis model is trained based on the dedicated training sequences corresponding to the one or more encoded representations, and the speech synthesis model is configured to convert input text into corresponding speech signals.

2. The method of claim 1, wherein generating a dedicated training sequence related to the target coded representation based on the target coded representation and reference speech comprises: Obtain identification information used to identify the reference speech; as well as The dedicated training sequence is generated based on the identifier information of the reference speech, the target encoding representation, and the reference speech.

3. The method according to claim 2, wherein the identification information is a training noise signal for the reference speech, and generating the dedicated training sequence comprises: The dedicated training sequence is generated by concatenating the target encoded representation, the training noise signal, and the reference speech.

4. The method of claim 1, wherein determining one or more encoded representations corresponding to one or more target texts in the target text set comprises: Obtain a set of phoneme sequences related to a target text set, wherein the phoneme sequences in the phoneme sequence set correspond to the target text in the target text set; as well as An encoding operation is performed on one or more target phoneme sequences in the phoneme sequence set to generate the one or more encoded representations.

5. The method according to claim 1, wherein training the speech synthesis model based on the dedicated training sequences corresponding to each of the one or more encoded representations comprises: Phoneme prediction is performed on the dedicated training sequence to generate a predicted phoneme sequence; Based on the predicted phoneme sequence, predictive acoustic features are generated; as well as The speech synthesis model is trained based on the predicted acoustic features.

6. The method according to claim 5, wherein training the speech synthesis model based on the predicted acoustic features comprises: Determine the given target text in the target text set that corresponds to the predicted acoustic features; Based on the given target text and the feature type of the predicted acoustic features, determine the reference acoustic features corresponding to the given target text; The training loss is determined based on the difference between the reference acoustic features and the predicted acoustic features; as well as The parameters of the speech synthesis model are updated based on the training loss.

7. The method of claim 5, wherein performing phoneme prediction on the dedicated training sequence to generate a predicted phoneme sequence comprises: Phoneme prediction is performed on the dedicated training sequence to obtain predicted fundamental frequency information for representing speech emotion and prosody, predicted duration information for representing phoneme duration, and predicted energy information for representing volume information. as well as The predicted phoneme sequence is determined based on the predicted fundamental frequency information, the predicted duration information, and the predicted energy information.

8. The method according to claim 1, further comprising: In response to the completion of training of the speech synthesis model, the inference text input to the speech synthesis model is obtained; Generate noise signals related to the inference text; as well as Based on the noise signal and the inferred text, the trained speech synthesis model is used to generate the speech signal corresponding to the inferred text.

9. An apparatus for speech synthesis, comprising: The extraction module is configured to extract one or more encoded representations corresponding to one or more target texts in a target text set; The generation module is configured to generate a dedicated training sequence for each of the one or more coding representations, based on each coding representation and a reference speech. as well as The training module is configured to train a speech synthesis model based on dedicated training sequences corresponding to the one or more encoded representations, wherein the speech synthesis model is configured to convert input text into corresponding speech signals.

10. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processor.

11. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images