Method, apparatus, device and storage medium for text-to-speech conversion

Abstract

According to embodiments of this disclosure, a method, apparatus, device, and storage medium for text-to-speech conversion are provided. The method includes generating a predicted speech representation of the target text read by a first speaker, based on a target text to be converted and a first timbre of a first speaker. The predicted speech representation indicates speech features that vary over time. The method further includes generating a predicted time-frequency representation of the target text read by a second speaker, based on the predicted speech representation and a second timbre of a second speaker. The predicted time-frequency representation indicates the speech signal strength that varies over time at different frequencies. The method further includes converting the predicted time-frequency representation into audio of the target text read by the second speaker. This reduces the difficulty of prediction and improves the sound quality of the generated audio.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and in particular to methods, apparatuses, devices, and computer-readable storage media for text-to-speech conversion. Background Technology

[0002] Text-to-speech (TTS) is a speech synthesis technology that converts text into natural-speech output. TTS has a wide range of applications. For example, it can help visually impaired people read text information or increase the readability of text documents. In some TTS scenarios, it is necessary to speak a passage of text in the voice of a real speaker to increase realism. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for text-to-speech conversion is provided. The method includes: generating a predicted speech representation of the target text read by a first speaker, based on a target text to be converted and a first speaker's timbre. The predicted speech representation indicates speech features that vary over time. The method further includes: generating a predicted time-frequency representation of the target text read by a second speaker, based on the predicted speech representation and a second speaker's timbre. The predicted time-frequency representation indicates the speech signal intensity that varies over time at different frequencies. The method further includes: converting the predicted time-frequency representation into audio of the target text read by the second speaker.

[0004] In a second aspect of this disclosure, a method for text-to-speech conversion is provided. The method includes: generating a first target speech representation of the first reference text read aloud by a first speaker, based on a first audio recording of the first reference text read aloud by a first speaker. The first target speech representation indicates speech features that vary over time. The method further includes: converting the first audio recording into a first target time-frequency representation of the first reference text read aloud by the first speaker. The first target time-frequency representation indicates the speech signal intensity that varies over time at different frequencies. The method further includes: training a first decoder based on the first target speech representation, the first target time-frequency representation, and a first timbre of the first speaker. The first decoder is used to convert the speech representation into a time-frequency representation in text-to-speech conversion.

[0005] In a third aspect of this disclosure, an apparatus for text-to-speech conversion is provided. The apparatus includes: a speech representation generation module configured to generate a predicted speech representation of the target text read by a first speaker, based on a target text to be converted and a first timbre of a first speaker, the predicted speech representation indicating speech features varying over time. The apparatus further includes: a time-frequency representation generation module configured to generate a predicted time-frequency representation of the target text read by a second speaker, based on the predicted speech representation and a second timbre of a second speaker, the predicted time-frequency representation indicating speech signal strength varying over time at different frequencies. The apparatus further includes: a conversion module configured to convert the predicted time-frequency representation into audio of the target text read by the second speaker.

[0006] In a fourth aspect of this disclosure, an apparatus for text-to-speech conversion is provided. The apparatus includes: a target feature extraction module configured to generate a first target speech representation of the first reference text read aloud by a first speaker, the first target speech representation indicating speech features that vary over time, based on a first audio recording of the first reference text read aloud by a first speaker. The apparatus further includes: an audio conversion module configured to convert the first audio recording into a first target time-frequency representation of the first reference text read aloud by the first speaker, the first target time-frequency representation indicating speech signal intensity that varies over time at different frequencies. The apparatus further includes: a decoder training module configured to train a first decoder based on the first target speech representation, the first target time-frequency representation, and a first timbre of the first speaker, the first decoder being used to convert the speech representation into a time-frequency representation in text-to-speech conversion.

[0007] In a fifth aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0008] In a sixth aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the second aspect.

[0009] In a seventh aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0010] In an eighth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the second aspect.

[0011] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0013] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0014] Figure 2A A schematic diagram of an example acoustic model according to some embodiments of the present disclosure is shown;

[0015] Figure 2B A schematic diagram of another example of an acoustic model according to some embodiments of the present disclosure is shown;

[0016] Figure 3 A schematic diagram illustrating the training of an acoustic model according to some embodiments of the present disclosure is shown;

[0017] Figure 4 A schematic diagram illustrating the generation of audio corresponding to target text according to some embodiments of the present disclosure is shown;

[0018] Figure 5 A flowchart illustrating a text-to-speech conversion process according to some embodiments of the present disclosure is shown;

[0019] Figure 6 A flowchart illustrating the process of training an acoustic model according to some embodiments of the present disclosure is shown;

[0020] Figure 7 A block diagram of an apparatus for text-to-speech conversion according to some embodiments of the present disclosure is shown;

[0021] Figure 8 A block diagram of an apparatus for training an acoustic model according to some embodiments of the present disclosure is shown; and

[0022] Figure 9 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0024] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0025] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0027] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0030] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can include different types of processing units or networks. As used in this paper, "determining the parameters of a model" or similar expressions refers to determining the values ​​of the model's parameters (also known as parameter values), including specific values, sets of values, or ranges of values.

[0031] Example Environment

[0032] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, an acoustic model 120 and a vocoder 130 for TTS are deployed in a computing device 110. The acoustic model 120 and the vocoder 130 are configured together to generate audio 102 corresponding to target text 101. Audio 102 simulates a speaker 150 (also referred to as a "second speaker") speaking the target text 101.

[0033] Acoustic model 120 is configured to generate a predicted time-frequency representation 103 of the target text 101 read out by speaker 150, which indicates the intensity of the speech signal over time at different frequencies. In other words, the predicted time-frequency representation 103 indicates the predicted speech intensity over time during the period when speaker 150 reads out the target text 101.

[0034] In embodiments of this disclosure, the predicted time-frequency representation 103 may include any suitable spectral features, such as Mel spectrum, inverse Mel spectrum coefficients (MFCC), etc. The acoustic model 120 is used to inject the timbre, rhythm, emotion, etc., inherent in the speech into the input target text 101.

[0035] The vocoder 130 is configured to convert the predicted time-frequency representation 103 into audio 102 of the target text 101 read aloud by the speaker 150. In other words, the vocoder 130 is used to convert the time-frequency representation 103 into an acoustic waveform. In embodiments of this disclosure, the vocoder 130 may have any suitable network, such as WaveNet, WaveRNN, etc.

[0036] In environment 100, computing device 110 can be any type of computing-capable device, including terminal devices or server devices. Terminal devices can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Server devices may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0037] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0038] As briefly mentioned earlier, TTS (Text-to-Speech) has been widely applied. The desired quality is high naturalness and sound reproduction in the generated audio, which is significantly influenced by the acoustic model. End-to-end acoustic models based on deep learning have been proposed. However, such models require extensive training data to achieve high naturalness. In non-autoregressive frameworks, the generated audio quality is often insufficient, easily containing background noise, static, and other interference.

[0039] Furthermore, such end-to-end acoustic models struggle to decouple timbre from other features (e.g., language, prosody, etc.). This limits the applicability of TTS. For instance, such acoustic models typically perform poorly in cross-linguistic TTS tasks.

[0040] On the other hand, self-supervised speech representation learning can learn meaningful speech representations from unlabeled data. In timbre conversion tasks, self-supervised speech representations have excellent feature decoupling capabilities.

[0041] Embodiments of this disclosure propose a scheme for TTS. According to various embodiments of this disclosure, the acoustic model for TTS includes a feature extractor and a decoder, wherein the feature extractor generates a predicted speech representation of a target text read by a first speaker, and the decoder generates a predicted time-frequency representation of the target text read by a second speaker based on the predicted speech representation and the timbre of a second speaker. During the training of the acoustic model, the feature extractor and decoder are trained separately using the target speech representation generated by a self-supervised model, without requiring end-to-end training of the acoustic model.

[0042] Training an acoustic model using self-supervised speech features brings additional acoustic information to the model. In this way, the prediction difficulty of the model can be reduced, and the quality of the generated audio can be improved. On the other hand, since self-supervised speech features can provide speech information independent of the speaker's timbre, using self-supervised speech features as intermediate features can achieve timbre decoupling. This enables the acoustic model to be used for cross-speaker and cross-language TTS tasks.

[0043] Overall architecture of acoustic model

[0044] Figure 2A An example of an acoustic model 120 according to some embodiments of the present disclosure is shown. Generally, the acoustic model 120 includes a feature extractor 201 and a first decoder 210. The feature extractor 201 is configured to generate a predicted speech representation of the speaker reading the text based on the text and the speaker's timbre. This predicted speech representation is an implicit feature representation and indicates time-varying speech features. This predicted speech representation indicates the time-varying speech features during the speaker reading the text. Through the following description of model training, it will be understood that this speech representation may include prosodic features, etc., and does not include timbre features or includes as few timbre features as possible. Figure 3 It will be understood that this speech representation may include prosodic features, etc., and does not include timbre features or includes as few timbre features as possible.

[0045] Timbre is used to identify a speaker and depends only on the speaker. Therefore, timbre can distinguish one speaker from other speakers. Timbre can be represented, for example, as a speaker's timbre identifier (ID), or a speaker's timbre vector.

[0046] In some embodiments, as Figure 2A shown, the feature extractor 201 may include a text encoder 240, a duration predictor 230, and a second decoder 220. Before being input into the text encoder 240, or within the text encoder 240, the processed text is divided into a sequence of speech units. In the sequence of speech units, each speech unit is arranged according to its position in the text. In the embodiments of the present disclosure, the speech unit can have any suitable granularity, such as words, phonemes, etc. Accordingly, the sequence of speech units can be a word sequence, a phoneme sequence, etc. As an example, taking words as speech units, the text "The weather is nice" can be divided into a sequence of speech units consisting of "天", "气", "不", "错".

[0047] The text encoder 240 is configured to generate a sequence of semantic representations of the text. The semantic representations in the sequence of semantic representations correspond one-to-one with the speech units of the text, and the position of each semantic representation in the sequence of semantic representations depends on the position of the corresponding speech unit in the text. That is, the sequence of semantic representations is a vectorized embedding of the speech units in the order of the speech units. For example, the sequence of semantic representations of the text "The weather is nice" includes the semantic representation W 天 of "tian", the semantic representation W 气 of "qi", the semantic representation W 不 of "bu", and the semantic representation W 错 of "cuo".

[0048] The duration predictor 230 receives the sequence of semantic representations and the voice color of the speaker. The duration predictor 230 is configured to predict the duration of each speech unit and update the received sequence of semantic representations according to the predicted duration. In the updated sequence of semantic representations output by the duration predictor 230, the number of semantic representations corresponding to each speech unit matches the predicted duration of the corresponding speech unit. For example, taking a frame as the time unit, if the duration predictor 230 predicts that "tian", "qi", "bu", and "cuo" last for 2 frames, 1 frame, 1 frame, and 3 frames respectively, the updated sequence of semantic representations includes 2 Ws arranged in sequence 天 , 1 W 气 , 1 W 不 , and 3 Ws 错 .

[0049] The second decoder 220 receives the updated sequence of semantic representations and the voice color of the speaker, and is configured to generate a predicted speech representation of the speaker reading out the text based on the updated sequence of semantic representations and the voice color of the speaker. For example, the predicted speech representation includes speech features arranged in a frame sequence to represent the change of speech features over time when the speaker reads out the text.

[0050] Figure 2A Only one example of the feature extractor 201 is shown. Other networks or modules can also be used to implement the feature extractor 201. In some embodiments, the feature extractor 201 may only include one network or sub-model to generate a predicted speech representation of the speaker reading out the text.

[0051] In some embodiments, the functions implemented by the duration predictor 230 and the text encoder 240 can be implemented by one network or sub-model. Figure 2B Another example of the acoustic model 120 is shown. In Figure 2BIn the example, feature extractor 201 includes a second decoder 220 and an encoder 250. Encoder 250 is configured to generate a semantic representation sequence of the text read by the speaker based on the text and the speaker's timbre. That is, encoder 250 implements the reference... Figure 2A The functions of the duration predictor 230 and the text encoder 240 are described.

[0052] The first decoder 210 receives the predicted speech representation generated by the feature extractor 201 and is configured to generate a predicted time-frequency representation based on the predicted speech representation and the speaker's timbre. For example, the first decoder 210 can generate a predicted Mel spectrum. The timbre used by the first decoder 210 and the timbre used by the feature extractor 201 can be the same or different, as will be referred to below. Figure 4 Described.

[0053] Training of acoustic models

[0054] Figure 3 A schematic diagram of the training of an acoustic model 120 according to some embodiments of the present disclosure is shown. Generally, the first encoder 210 and the feature extractor 201 can be trained simultaneously or separately.

[0055] Audio 312, in which speaker 310 reads the reference text 316, is any audio from the training dataset. A target speech representation 314 of speaker 310 reading the reference text 316 is generated from audio 312 using feature extraction model 301. Feature extraction model 301 can be any suitable unsupervised or self-supervised speech representation learning model, such as Hubert, Wav2vec, etc.

[0056] The target speech representation 314 indicates speech features that change over time. That is, the target speech representation 314 indicates speech features that change over time during the period when the speaker 310 reads the reference text 316. For example, the target speech representation 314 may include speech features arranged in a frame sequence.

[0057] A target time-frequency representation 313 is generated based on audio 312, in which speaker 310 reads reference text 316. The target time-frequency representation 313 indicates the speech signal intensity varying over time at different frequencies. That is, the target time-frequency representation 313 indicates the speech signal intensity varying over time during the period when speaker 310 reads reference text 316. For example, the target time-frequency representation 313 could be the Mel spectrum of the speech text 316 read by speaker 310. Audio 312 can be converted into the corresponding target time-frequency representation 313 in any suitable manner, and the scope of this disclosure is not limited in this respect.

[0058] The target speech representation 314, the target time-frequency representation 313, and the speaker's timbre 311 are used to train the first decoder 210. During training, L1 loss can be used as the loss function.

[0059] As an example, a target speech representation 314 can be fed into a first decoder 210. The first decoder 210 generates a predicted time-frequency representation 323 of the reference text 316 read aloud by the speaker 310 based on the target speech representation 314. The predicted time-frequency representation 323 indicates the speech signal strength predicted by the first decoder 210 as changing over time during the speaker 310's reading of the reference text 316. Based on the difference between the predicted time-frequency representation 323 and the target time-frequency representation 313, parameters of the first decoder 210 can be determined. For example, an L1 loss can be determined based on the difference between the predicted time-frequency representation 323 and the target time-frequency representation 313 to update the parameters of the first decoder 210.

[0060] In addition to the target time-frequency representation 313 and the target speech representation 314, reference text 316 and target duration 315 can also be extracted from audio 312. Target duration 315 is the duration of each speech unit in reference text 316 in audio 312. Feature extractor 201 is trained using target speech representation 314, target duration 315, reference text 316, and the timbre 311 of speaker 310. During training, L1 loss can be used as the loss function.

[0061] by Figure 2A The training of the feature extractor 201 is described using an example implementation of the feature extractor 201 shown. Reference text 316, or preprocessed (e.g., segmented into speech units) reference text 316, is input to text encoder 240. Text encoder 240 generates a sequence of semantic representations of the reference text 316. Each semantic representation in this sequence corresponds one-to-one with a speech unit in the reference text 316, and the position of each semantic representation in the sequence depends on the position of its corresponding speech unit in the reference text 316.

[0062] The semantic representation sequence generated by text encoder 240 is input to duration predictor 230 after being combined (e.g., added) with timbre 311. Duration predictor 230 generates a predicted duration for each speech unit in reference text 316. The difference between the predicted duration and the target duration 315 can be used to determine the loss function for training, such as L1 loss.

[0063] Based on the predicted duration, the duration predictor 230 further updates the semantic representation sequence of the reference text 316. In the updated semantic representation sequence, the number of semantic representations corresponding to each speech unit matches the predicted duration of the corresponding speech unit.

[0064] The second decoder 220 receives the updated semantic representation sequence and the timbre 311 of the speaker 310, and generates a predicted speech representation 324 of the reference text 316 read by the speaker 310. The predicted speech representation 324 indicates the predicted speech features that change over time during the speaker 310's reading of the reference text 316. The difference between the predicted speech representation 324 and the target speech representation 314 can be used to determine the loss function for training, such as L1 loss.

[0065] The feature extractor 201 is trained end-to-end as a whole. The feature extractor 201 and the first decoder 210 can use the same training dataset but are trained separately.

[0066] Given the characteristics of unsupervised or self-supervised representation learning, the target speech representation 314 generated by the feature extraction model 301 characterizes linguistic information independent of the speaker's timbre. Therefore, during training, the feature extractor 201 learns how to extract speech features independent of the speaker's timbre. Thus, by using unsupervised or self-supervised features as intermediate features, timbre decoupling capability can be achieved. The acoustic model 120 obtained in this way can achieve text-to-speech conversion across speakers and across languages.

[0067] On the other hand, using unsupervised or self-supervised features as intermediate features adds extra information to the acoustic model. This can reduce the difficulty of prediction and improve the sound quality of the generated audio.

[0068] Furthermore, audio 102 in any language can be used in the training of the acoustic model 120. In some embodiments, the training dataset may include audio 102 in different languages.

[0069] Application of acoustic models in TTS

[0070] The following is for reference. Figure 4 This describes how acoustic model 120 is used in TTS tasks. For example... Figure 4 As shown, feature extractor 201 generates a predicted speech representation 401 of the first speaker 410 speaking the target text 101 based on the target text 101 to be converted and the first timbre 411 of the first speaker 410. The predicted speech representation 401 indicates the predicted speech features that change over time during the first speaker 410 speaking the target text 101. For example, the predicted speech representation 401 may include speech features arranged in a frame sequence.

[0071] The following is based on Figure 2AThe structure of the feature extractor 201 shown is used as an example to describe an example of generating a predicted speech representation 401. Before being input to the text encoder 240, or during the text encoder 240, the target text 101 is divided into a sequence of speech units. In the speech unit sequence, the individual speech units (e.g., words, phonemes) are arranged in the order they appear in the target text 101. The text encoder 240 generates a semantic representation sequence 402 of the target text 101, which is also called a second semantic representation sequence. The semantic representations in the semantic representation sequence 402 correspond one-to-one with the speech units of the target text 101, and the position of each semantic representation in the semantic representation sequence depends on the position of the corresponding speech unit in the target text 101.

[0072] The first timbre 411 of the first speaker 410 (e.g., timbre ID or timbre vector) is combined with the semantic representation sequence 402, for example, by addition or concatenation. The combined speech representation sequence 402 and the first timbre 411 are fed to the duration predictor 230. The duration predictor 230 determines the predicted duration of each speech unit in the target text 101 based on the first timbre 411 and the semantic representation sequence 402. It is understood that this predicted duration is generated according to the prosodic features of the first speaker 410.

[0073] The duration predictor 230 further updates the semantic representation sequence based on the predicted duration. In the updated semantic representation sequence 403, the number of semantic representations corresponding to each speech unit matches the predicted duration of that speech unit. For example, suppose a semantic representation vector corresponds to one frame in time, and the predicted duration of a certain speech unit is 3 frames. In the updated semantic representation sequence 403, the number of semantic representations corresponding to that speech unit is 3.

[0074] The updated semantic representation sequence 403 and the first timbre 411 are input to the second decoder 220. Accordingly, the second decoder 220 generates a predicted speech representation 401 for the target text 101 read by the first speaker 410, which indicates the speech features that change over time. The predicted speech representation 401 indicates the predicted speech features that change over time during the period when the first speaker 410 reads the target text 101.

[0075] As referenced above Figure 3 As described, the training of the feature extractor 201 is supervised using unsupervised or self-supervised speech features. Therefore, in the application of the acoustic model 120, the predicted speech representation 401 generated by the feature extractor 201 includes prosodic features of the first speaker 410, but includes little or no timbre features. The speech features in the predicted speech representation 401 are decoupled from timbre.

[0076] The first decoder 210 generates a predicted time-frequency representation 103 for the target text 101 read by the second speaker 150, based on the predicted speech representation 401 and the second timbre 421 of the second speaker 150. The predicted time-frequency representation 103 indicates the speech signal strength over time at different frequencies. That is, the predicted time-frequency representation 103 indicates the predicted speech signal strength over time at different frequencies during the period when the second speaker 150 reads the target text 101. For example, the predicted time-frequency representation 103 may include a predicted Mel spectrum.

[0077] The second timbre 421 of the second speaker 150 is fused into the predicted speech representation 401, which originally lacked timbre features, by the first decoder 210. In this way, the generated predicted time-frequency representation 103 can reflect the prosodic features of the first speaker 410 and the timbre features of the second speaker 150.

[0078] The vocoder 130 converts the predicted time-frequency representation 103 into audio 102 of the target text 101 read aloud by the second speaker 150. The audio 102 reflects the rhythm of the first speaker 410 and the timbre of the second speaker 150.

[0079] In order to achieve reference Figure 4 The described process involves acoustic model 120 learning the timbre of a first speaker 410 and a second speaker 150 through a training process. Specifically, the training dataset for acoustic model 120 includes audio recordings of the first speaker reading the reference text in a first language and the second speaker reading the reference text in a second language. In this paper, the speech used by the speakers in the audio of the training dataset is referred to as the reference language. Acoustic model 120 has learned the features of the first speaker 410 speaking the first language and the features of the second speaker speaking the second language.

[0080] In some embodiments, the first speaker 410 is the same as the second speaker 150. In this case, the first timbre 411 is the same as the second timbre 421.

[0081] In some embodiments, the acoustic model 120 can be used for cross-speaker TTS. The first speaker 410 is different from the second speaker 150, and the first timbre 411 is different from the second timbre 412.

[0082] In some embodiments, the acoustic model 120 can be used for cross-language TTS. The target text 101 is in the first language spoken by the first speaker 410. The second language spoken by the second speaker 150 is different from the first language. The second speaker 150 may not have the ability to speak the first language, or the acoustic model 120 may not have learned how the second speaker 150 speaks the first language. For example, the training of the acoustic model 120 uses audio of the first speaker 410 speaking Chinese and audio of the second speaker 150 speaking Japanese. The target text 101 includes Chinese. In this case, the acoustic model 120 can be used to generate audio of the second speaker 150, who does not originally speak Chinese, reading the target text 101 in Chinese using their own timbre. Cross-language TTS is implemented using embodiments of this disclosure.

[0083] Example process

[0084] Figure 5 A flowchart of a text-to-speech conversion process 500 according to some embodiments of the present disclosure is shown. Process 500 can be implemented at a computing device 110. Reference is made below. Figure 1 , Figure 2A , Figure 2B and Figure 4 To describe process 500.

[0085] In box 510, computing device 110 generates a predicted speech representation 401 of the target text 101 read by the first speaker 410, based on the target text 101 to be converted and the first timbre 411 of the first speaker 410. The predicted speech representation 401 indicates speech features that change over time. The predicted speech representation 401 can be generated by feature extractor 201.

[0086] In box 520, computing device 110 generates a predicted time-frequency representation 103 of the target text 101 read by the second speaker 150, based on predicted speech representation 401 and the second timbre 412 of the second speaker 150. The predicted time-frequency representation 103 indicates the speech signal strength varying over time at different frequencies. The predicted time-frequency representation 103 can be generated by the first decoder 210.

[0087] In some embodiments, a first semantic representation sequence of the target text 101 can be generated based on the target text 101 and the first timbre 411. The number of semantic representations corresponding to speech units in the target text in the first semantic representation sequence matches the predicted duration of the speech units. The first semantic representation sequence is, for example, an updated semantic representation sequence 403. A predicted speech representation 401 can be generated based on the first semantic representation sequence and the first timbre 411. For example, it can be generated by… Figure 2B The encoder 250 generates the first semantic representation sequence.

[0088] In some embodiments, to generate a first semantic representation sequence, a second semantic representation sequence of the target text 101 can be generated based on the target text 101. The second semantic representation sequence includes semantic representations corresponding one-to-one with the speech units. For example, the second semantic representation sequence is... Figure 4 The semantic representation sequence 402 is used. The predicted duration of the speech unit can be determined based on the second semantic representation sequence and the first timbre 411. Based on the predicted duration, the second semantic representation is updated to the first semantic representation sequence.

[0089] In box 530, computing device 110 converts the predicted time-frequency representation 103 into audio 102 of the target text 101 read aloud by second speaker 150. The predicted time-frequency representation 103 can be converted into audio 102 by vocoder 130.

[0090] In some embodiments, the target text 101 includes text in a first language, and the reference language of the first speaker 410 and the second speaker 150 is the first language. In some embodiments, the first speaker 410 is different from the second speaker 150. In some embodiments, the first speaker 410 is the same as the second speaker 150.

[0091] In some embodiments, the target text 101 includes text in a first language, the reference language of the first speaker 410 is the first language, and the reference language of the second speaker 150 is a second language different from the first language. That is, in the training dataset used for the acoustic model, the first speaker 410 uses the first language, while the second speaker 150 uses the second language.

[0092] Figure 6 A flowchart of a process 600 for training an acoustic model according to some embodiments of the present disclosure is shown. Process 600 can be implemented at a computing device, which may be the same as or different from computing device 110. The following... Figure 3 To describe process 600.

[0093] In box 610, the computing device generates a first target speech representation of the first reference text read by a first speaker, based on the first audio of the first reference text read by the first speaker. The first target speech representation indicates speech features that change over time. For example, a self-supervised or unsupervised feature extraction model 301 can be used to generate the target speech representation 314 based on the audio 312. The first speaker can be any speaker involved in the training dataset.

[0094] In box 620, the computing device converts the first audio into a first target time-frequency representation of the first reference text read aloud by the first speaker. The first target time-frequency representation indicates the intensity of the speech signal at different frequencies over time.

[0095] In box 630, the computing device trains a first decoder 210 based on a first target speech representation, a first target time-frequency representation, and a first speaker's timbre. The first decoder 210 is used to convert the speech representation into a time-frequency representation in text-to-speech conversion.

[0096] In some embodiments, a predicted time-frequency representation of the first reference text read by the first speaker can be generated by the first decoder 410 based on the first target speech representation and the first timbre. The predicted time-frequency representation indicates the speech signal intensity at different frequencies over time. Parameters of the first encoder can be determined based on the difference between the predicted time-frequency representation and the first target time-frequency representation.

[0097] In some embodiments, a second target speech representation of the second reference text read by the second speaker can be generated based on the second audio of the second reference text read by the second speaker. This second target speech representation indicates speech features that vary over time. The second speaker is different from the first speaker, and the first and second reference texts are texts in different languages. The second audio can be converted into a second target time-frequency representation of the second reference text read by the second speaker. This second target time-frequency representation indicates the speech signal intensity that varies over time at different frequencies. The first decoder can be trained based on the second target speech representation, the second target time-frequency representation, and the second timbre of the second speaker. The training of the acoustic model 120 can be cross-linguistic.

[0098] In some embodiments, the target duration of the speech unit in the first reference text in the first audio can also be determined. The text encoder 240, duration predictor 230, and second decoder 220 can be trained based on the first reference text, the first timbre, the target duration, and the target speech representation. The text encoder 240, duration predictor 230, and second decoder 220 are configured together to generate a speech representation of the text read by the speaker.

[0099] Example devices and equipment

[0100] Figure 7 A schematic structural block diagram of an apparatus 700 for text-to-speech conversion according to certain embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in a computing device 110. Various modules / components in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0101] As shown in the figure, the device 700 includes a speech representation generation module 710, configured to generate a predicted speech representation of the target text read by the first speaker based on the target text to be converted and a first speaker's first timbre, wherein the predicted speech representation indicates speech features that change over time. The device 700 also includes a time-frequency representation generation module 720, configured to generate a predicted time-frequency representation of the target text read by the second speaker based on the predicted speech representation and a second speaker's second timbre, wherein the predicted time-frequency representation indicates the speech signal strength that changes over time at different frequencies. The device 700 further includes a conversion module 730, configured to convert the predicted time-frequency representation into audio of the target text read by the second speaker.

[0102] In some embodiments, the speech representation generation module 710 includes: an encoding module configured to generate a first semantic representation sequence of the target text based on the target text and a first timbre, wherein the number of semantic representations corresponding to speech units in the target text in the first semantic representation sequence matches the predicted duration of the speech units; and a decoding module configured to generate a predicted speech representation based on the first semantic representation sequence and the first timbre.

[0103] In some embodiments, the encoding module includes: a text encoding module configured to generate a second semantic representation sequence of the target text based on the target text, the second semantic representation sequence including semantic representations corresponding one-to-one with speech units; a duration prediction module configured to determine the predicted duration of the speech units based on the second semantic representation sequence and a first timbre; and an encoding update module configured to update the second semantic representation to the first semantic representation sequence based on the predicted duration.

[0104] In some embodiments, the target text includes text in a first language, and the reference language of the first speaker and the second speaker is the first language.

[0105] In some embodiments, the first speaker is different from the second speaker.

[0106] In some embodiments, the target text includes text in a first language, the reference language of the first speaker is the first language, and the reference language of the second speaker is a second language different from the first language.

[0107] Figure 8 A schematic structural block diagram of an apparatus 800 for training acoustics according to certain embodiments of the present disclosure is shown. The apparatus 800 may be implemented as or included in a computing device. The various modules / components in the apparatus 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0108] As shown in the figure, the device 800 includes a target feature extraction module 810, configured to generate a first target speech representation of the first reference text read by a first speaker, based on a first audio recording of the first reference text read by a first speaker. The first target speech representation indicates speech features that change over time. The device 800 also includes an audio conversion module 820, configured to convert the first audio recording into a first target time-frequency representation of the first reference text read by the first speaker. The first target time-frequency representation indicates the speech signal intensity that changes over time at different frequencies. The device 800 further includes a training module 830, configured to train a first decoder based on the first target speech representation, the first target time-frequency representation, and a first timbre of the first speaker. The first decoder is used to convert the speech representation into a time-frequency representation in text-to-speech conversion.

[0109] In some embodiments, the training module 830 is further configured to: generate a predicted time-frequency representation of a first reference text read by a first speaker based on a first target speech representation and a first timbre, according to a first decoder, wherein the predicted time-frequency representation indicates the speech signal intensity that varies over time at different frequencies; and determine parameters of a first encoder based on the difference between the predicted time-frequency representation and the first target time-frequency representation.

[0110] In some embodiments, the target feature extraction module 810 is further configured to generate a second target speech representation of the second reference text read by the second speaker, based on the second audio of the second reference text read by the second speaker. The second target speech representation indicates speech features that vary over time, wherein the second speaker is different from the first speaker, and the first reference text and the second reference text are texts in different languages. The audio conversion module 820 is further configured to convert the second audio into a second target time-frequency representation of the second reference text read by the second speaker. The second target time-frequency representation indicates the speech signal intensity that varies over time at different frequencies. The decoder training module 830 is further configured to train a first decoder based on the second target speech representation, the second target time-frequency representation, and the second timbre of the second speaker.

[0111] In some embodiments, the apparatus 800 further includes: a duration determination module configured to determine a target duration of a speech unit in a first reference text in a first audio; and an extractor training module configured to train a text encoder, a duration predictor, and a second decoder based on the first reference text, a first timbre, the target duration, and a target speech representation, wherein the text encoder, the duration predictor, and the second decoder are together configured to generate a speech representation of a speaker reading text.

[0112] Figure 9 A block diagram illustrating a computing device 900 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 9The computing device 900 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 9 The computing device 900 shown can be used to implement Figure 1 The computing device 110.

[0113] like Figure 9 As shown, computing device 900 is in the form of a general-purpose computing device. Components of computing device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 920. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 900.

[0114] Computing device 900 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to computing device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 900.

[0115] The computing device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 9 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 920 may include computer program product 925 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0116] The communication unit 940 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 900 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0117] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 900 can also communicate as needed with one or more external devices (not shown) via communication unit 940. These external devices, such as storage devices, display devices, etc., can communicate with one or more devices that enable user interaction with computing device 900, or with any device (e.g., network card, modem, etc.) that enables computing device 900 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0118] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0119] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0121] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0123] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for text-to-speech conversion, comprising: generating, using a trained feature extractor, a predicted speech representation of a first speaker reading out a target text based on the target text to be converted and a first vocal color of the first speaker, the first vocal color comprising a vocal color identifier or a vocal color vector, the predicted speech representation indicating speech features varying over time and removing vocal color features, the feature extractor being trained using unsupervised or self-supervised speech features; generating, using a first decoder, a predicted time-frequency representation of a second speaker reading out the target text based on the predicted speech representation and a second vocal color of the second speaker, the predicted time-frequency representation indicating speech signal intensity varying over time at different frequencies; and converting the predicted time-frequency representation into an audio of the second speaker reading out the target text.

2. The method of claim 1, wherein generating the predicted speech representation comprises: generating, using the trained feature extractor, a first semantic representation sequence of the target text based on the target text and the first vocal color, a number of semantic representations corresponding to speech units in the target text in the first semantic representation sequence matching predicted durations of the speech units; and generating, using the trained feature extractor, the predicted speech representation based on the first semantic representation sequence and the first vocal color.

3. The method of claim 2, wherein generating the first semantic representation sequence comprises: generating, based on the target text, a second semantic representation sequence of the target text, the second semantic representation sequence comprising semantic representations one-to-one corresponding to the speech units; determining, based on the second semantic representation sequence and the first vocal color, predicted durations of the speech units; and updating, based on the predicted durations, the second semantic representation sequence to the first semantic representation sequence.

4. The method of claim 1, wherein the target text comprises text in a first language, and a reference language of the first speaker and the second speaker is the first language.

5. The method of claim 4, wherein the first speaker is different from the second speaker.

6. The method of claim 1, wherein the target text comprises text in a first language, a reference language of the first speaker is the first language, and a reference language of the second speaker is a second language different from the first language.

7. A method for text-to-speech conversion, comprising: generating, using an unsupervised or self-supervised speech representation model, a first target speech representation of a first speaker reading out a first reference text based on a first audio of the first speaker reading out the first reference text, the first target speech representation indicating speech features varying over time and removing vocal color features. based on the first reference text and a first vocal color of the first speaker, the first vocal color comprising a vocal color identifier or a vocal color vector of the first speaker, generating, with a feature extractor, a predicted speech representation of the first speaker reading out the first reference text, the predicted speech representation indicating speech features over time; training the feature extractor based on a difference between the first target speech representation and the predicted speech representation; converting the first audio into a first target time-frequency representation of the first speaker reading out the first reference text, the first target time-frequency representation indicating speech signal intensity over time at different frequencies; and training, based on the first target speech representation, the first target time-frequency representation, and the first vocal color of the first speaker, a first decoder for converting speech representations generated by the feature extractor into time-frequency representations in text-to-speech conversion.

8. The method of claim 7, wherein training the first decoder comprises: generating, from the first decoder, a predicted time-frequency representation of the first speaker reading out the first reference text based on the first target speech representation and the first vocal color, the predicted time-frequency representation indicating speech signal intensity over time at different frequencies; and determining parameters of the first decoder based on a difference between the predicted time-frequency representation and the first target time-frequency representation.

9. The method of claim 7, further comprising: generating, based on second audio of a second speaker reading out a second reference text, a second target speech representation of the second speaker reading out the second reference text, the second target speech representation indicating speech features over time, wherein the second speaker is different from the first speaker and the first reference text and the second reference text are texts in different languages; converting the second audio into a second target time-frequency representation of the second speaker reading out the second reference text, the second target time-frequency representation indicating speech signal intensity over time at different frequencies; and training the first decoder based on the second target speech representation, the second target time-frequency representation, and a second vocal color of the second speaker.

10. The method of claim 7, further comprising: determining a target duration of a speech unit in the first reference text in the first audio; and training, based on the first reference text, the first vocal color, the target duration, and the first target speech representation, a text encoder, a duration predictor, and a second decoder, the text encoder, the duration predictor, and the second decoder together configured to generate speech representations of speakers reading out texts.

11. An apparatus for text-to-speech conversion, comprising: ​ a speech representation generation module configured to generate, based on a target text to be converted and a first vocal color of a first speaker, a predicted speech representation of the first speaker reading the target text using a trained feature extractor, the first vocal color comprising a vocal color identifier or a vocal color vector of the first speaker, the predicted speech representation indicating speech features varying over time and free of vocal color features, the feature extractor being trained using unsupervised or self-supervised speech features; a time-frequency representation generation module configured to generate, based on the predicted speech representation and a second vocal color of a second speaker, a predicted time-frequency representation of the second speaker reading the target text using a first decoder, the predicted time-frequency representation indicating speech signal intensity varying over time at different frequencies; and a conversion module configured to convert the predicted time-frequency representation into an audio of the second speaker reading the target text.

12. An apparatus for text-to-speech conversion, comprising: a target feature extraction module configured to generate, based on a first audio of a first speaker reading a first reference text, a first target speech representation of the first speaker reading the first reference text using an unsupervised or self-supervised speech representation model, the first target speech representation indicating speech features varying over time and free of vocal color features; a predicted speech generation module configured to generate, based on the first reference text and a first vocal color of the first speaker, a predicted speech representation of the first speaker reading the first reference text using a feature extractor, the first vocal color comprising a vocal color identifier or a vocal color vector of the first speaker, the predicted speech representation indicating speech features varying over time; a feature extractor training module configured to train the feature extractor based on a difference between the first target speech representation and the predicted speech representation; an audio conversion module configured to convert the first audio into a first target time-frequency representation of the first speaker reading the first reference text, the first target time-frequency representation indicating speech signal intensity varying over time at different frequencies; and a decoder training module configured to train a first decoder for converting speech representation into time-frequency representation in text-to-speech conversion based on the first target speech representation, the first target time-frequency representation, and the first vocal color of the first speaker.

13. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-6 or the method according to any one of claims 7-10.

14. A computer-readable storage medium having stored thereon a computer program executable by a processor to implement the method according to any one of claims 1-6 or the method according to any one of claims 7-10.

Images