A speech synthesis method, apparatus, device, and medium

Abstract

This application provides a speech synthesis method, apparatus, device, and medium, belonging to the field of computer technology. The method includes: encoding style prompt speech into discrete speech code using a pre-trained speech quantizer to obtain prompt speech code; using a pre-trained text-to-discrete speech coding module, based on a penalty-before-sampling strategy, performing multiple inferences according to the target text and the prompt text and prompt speech code corresponding to the style prompt speech to obtain the target discrete speech code corresponding to the target text; and using a pre-trained discrete speech code to waveform module, converting the target discrete speech code into target speech based on the timbre prompt speech.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speech synthesis method, apparatus, device, and medium. Background Technology

[0002] With the continuous development of deep learning and the popularization of intelligent devices, text-to-speech (TTS) technology has been widely used in many fields such as intelligent assistants, navigation systems, and educational tools.

[0003] In related technologies, a neural encoder model is typically used first to quantize the speech waveform into discrete speech codes, then an autoregressive language model (LM) is used for encoding and decoding modeling, and a random sampling strategy is used for inference to achieve speech synthesis.

[0004] However, the random sampling strategy can lead to instability issues such as missing words, repeated readings, and infinite loops, resulting in poor stability of synthesized speech. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide a speech synthesis method, apparatus, device, and medium to overcome or at least partially solve the above problems.

[0006] A first aspect of this application provides a speech synthesis method, the method comprising:

[0007] The style cue speech is encoded into discrete speech codes by using a pre-trained speech quantizer, thus obtaining the cue speech code.

[0008] By using a pre-trained text-to-discrete speech coding module, multiple inferences are performed based on the target text and the prompt text and prompt speech coding corresponding to the style prompt speech to obtain the target discrete speech coding corresponding to the target text.

[0009] The target discrete speech code is converted into target speech by a pre-trained discrete speech code-to-waveform module, based on the timbre-cue speech;

[0010] The target discrete speech code is determined through the following steps:

[0011] Based on the target text and the corresponding prompt text and prompt voice code s of the style prompt voice. :t′ By reasoning, the decoded code s is obtained. t′:t′+t-1 Where t represents the decoding step size;

[0012] If the current inference count is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is reduced by a factor of σ, where n represents the consecutive repetition threshold, w1 represents the window length, and σ represents the penalty coefficient.

[0013] If the current inference count is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is then reduced by a factor of σ, where w2 represents the window length;

[0014] Divide the current probability value (reduced by a factor of σ) by the temperature coefficient to obtain the updated probability distribution. Based on this updated probability distribution, decode the speech code s for the current decoding time using multinomial sampling. t And based on the speech code s t The inference and decoding steps are performed iteratively until the final encoding is obtained, thus obtaining the target discrete speech code.

[0015] A second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0016] The first processing module is used to encode the style prompt speech into discrete speech codes using a pre-trained speech quantizer, thereby obtaining the prompt speech code;

[0017] The second processing module is used to perform multiple inferences based on the target text and the prompt text and prompt speech code corresponding to the style prompt speech through a pre-trained text-to-discrete speech coding module to obtain the target discrete speech code corresponding to the target text.

[0018] The speech synthesis module is used to convert the target discrete speech code into target speech by using a pre-trained discrete speech code to waveform module, with timbre prompt speech as a condition;

[0019] Specifically, the second processing module is used to perform the following steps:

[0020] Based on the target text and the corresponding prompt text and prompt voice code s of the style prompt voice. :t′ By reasoning, the decoded code s is obtained. t′:t′+t-1 Where t represents the decoding step size;

[0021] If the current inference count is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is reduced by a factor of σ, where n represents the consecutive repetition threshold, w1 represents the window length, and σ represents the penalty coefficient.

[0022] If the current inference count is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is then reduced by a factor of σ, where w2 represents the window length;

[0023] Divide the current probability value (reduced by a factor of σ) by the temperature coefficient to obtain the updated probability distribution. Based on this updated probability distribution, decode the speech code s for the current decoding time using multinomial sampling. t And based on the speech code s t The inference and decoding steps are performed iteratively until the final encoding is obtained, thus obtaining the target discrete speech code.

[0024] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the speech synthesis method as described in the first aspect.

[0025] A fourth aspect of this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the speech synthesis method as described in the first aspect.

[0026] A fifth aspect of this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the speech synthesis method as described in the first aspect.

[0027] The embodiments of this application have the following advantages: Considering that random sampling strategies can lead to unstable phenomena such as missing words, repeated readings, and infinite loops, this application designs a penalty-before-sampling strategy to optimize the speech synthesis process. Specifically, when the current inference count is less than n, the obtained speech code is... The probability value at the corresponding index is reduced by a factor of σ to decrease the interference of the inference result caused by the variable-length silence segment at the end of the prompt speech at the beginning of inference; when the current number of inferences is not less than n, the obtained speech code is... The probability value at the corresponding index is reduced by a factor of σ to ensure that the decoded speech code has the same distribution as the real speech code as possible; then, the speech code s at the current decoding time is decoded based on the temperature coefficient and polynomial sampling. tThe subsequent reasoning and decoding follow the aforementioned steps in a loop until the final encoding is obtained, resulting in the target discrete speech code. Finally, the target speech (i.e., synthesized speech) is obtained based on the target discrete speech code. This reduces the occurrence of unstable phenomena such as missing words, repeated readings, and infinite loops, thereby significantly improving the stability of synthesized speech. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating an implementation of a speech synthesis method in this application.

[0030] Figure 2 This is a schematic diagram of a reasoning process in an embodiment of this application;

[0031] Figure 3 This is a flowchart illustrating a penalty-then-sampling strategy in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of a VALL-EAR-like training process in a related technology;

[0033] Figure 5 This is a schematic diagram of the training process of a text-to-discrete speech coding module in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of a training process in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the structure of a speech synthesis device according to an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0037] To facilitate understanding of the technical solutions provided in this application, the main technical concepts involved in the embodiments of this application are briefly described below.

[0038] Text-to-speech (TTS): The goal is to convert input text into natural and fluent speech output.

[0039] Generative Pre-trained Transformer (GPT) model: a deep learning model based on autoregressive language model; the core idea of ​​autoregressive language model is to use the generated words to predict the probability of the next word. In GPT model, this autoregressive characteristic is reflected in the model generating the next word based on the given context until a certain length of text is generated or a specific end character is encountered.

[0040] VITS is an end-to-end text-to-speech architecture that uses variational inference, supplemented by a normalization process and adversarial training, to produce more natural audio.

[0041] Embedding layer: A special layer in a neural network used to map discrete input data (such as words, characters, or tokens) into a continuous vector space.

[0042] XLSR-53 model: It is a pre-trained model for 53 languages ​​that Facebook has open-sourced.

[0043] Mean Opinion Score (MOS): This is a measure of the average opinion people give to a particular service, product, or experience.

[0044] In recent years, with the continuous development of deep learning and the popularization of intelligent devices, TTS (Text-to-Speech) technology has been widely applied in many fields. For example, intelligent assistants (such as Siri and Xiao Ai) utilize TTS technology to provide a natural interactive experience, enabling users to easily obtain information and complete tasks through voice commands; navigation systems use voice prompts generated based on TTS technology to help drivers safely reach their destinations, reducing reliance on screens; educational tools use TTS technology to provide convenience for visually impaired and dyslexic individuals, enabling them to acquire knowledge more easily. Furthermore, the emergence of the GPT-4O interaction mode has significantly improved the human-computer interaction experience, and synthesized speech, as the direct entry point for human-computer interaction, directly determines the quality of the user experience through its natural fluency.

[0045] In related technologies, traditional TTS systems, constrained by training data and training paradigms, typically use Mel spectrograms as intermediate representations and cascade modules such as prosody prediction, glyph-to-phoneme conversion, acoustic models, and vocoders. These systems require high-quality studio recording data for training, but recording high-quality data is very limited and expensive. Furthermore, the speech synthesized by such systems rarely reaches human-level quality, primarily for the following reasons:

[0046] On the one hand, spontaneous human speech contains not only linguistic information but also a lot of subtle emotional paralinguistic information, while studio-recorded speech is usually reading or performing according to a pre-defined text, lacking diversity;

[0047] On the other hand, TTS systems trained with limited data generally have poor generalization ability and basically lack zero-sample synthesis capability. That is, for speakers they have never seen before, the naturalness and similarity of the synthesized speech generated by the TTS system will be severely reduced.

[0048] In contrast, large-scale TTS systems support training using tens of thousands of hours of data crawled from the Internet, which enables them to have better zero-shot synthesis capabilities.

[0049] Current speech synthesis methods based on large-scale TTS systems typically involve first quantizing the speech waveform into discrete speech codes using a neural encoder model, then using an autoregressive language model for encoding and decoding, and finally employing a random sampling strategy for inference to achieve speech synthesis. For example, VALL-E-based speech synthesis methods use an autoregressive (AR) language model to encode coarse-grained codes, a non-autoregressive (NAR) language model to encode the remaining fine-grained codes, and a random sampling strategy for inference.

[0050] However, due to the instability caused by random sampling strategies, such as missing words, repeated readings, and infinite loops, the synthesized speech obtained by current speech synthesis methods based on large-scale TTS systems has poor stability.

[0051] To address the problems existing in the aforementioned related technologies, this application proposes a speech synthesis method, apparatus, device, and medium. Based on a penalty-before-sampling strategy, the speech synthesis process is optimized, which can reduce the occurrence of unstable phenomena such as missing words, repeated readings, and infinite loops, thereby significantly improving the stability of synthesized speech.

[0052] The following description, in conjunction with the accompanying drawings, details a speech synthesis method, apparatus, device, and medium provided in this application through some embodiments and application scenarios.

[0053] Firstly, referring to Figure 1 The diagram shown is an implementation flowchart of a speech synthesis method provided in this application embodiment. The method includes the following steps:

[0054] Step S11: Encode the style cue speech into discrete speech codes using a pre-trained speech quantizer to obtain the cue speech codes.

[0055] The speech quantizer can be constructed based on a neural encoder model or a kmean model.

[0056] In specific implementation, refer to Figure 2 The illustrated inference process diagram shows that a speech quantizer is used to encode the style-based prompt speech into discrete speech codes, and the resulting discrete speech codes are then used as the prompt speech codes. The prompt speech codes, the target text (i.e., the input text), and the prompt text corresponding to the style-based prompt speech (i.e., the transcribed text) are then input into the text-to-discrete speech encoding module for further processing.

[0057] Step S12: Using a pre-trained text-to-discrete speech coding module, perform multiple inferences based on the target text and the prompt text and prompt speech coding corresponding to the style prompt speech to obtain the target discrete speech coding corresponding to the target text.

[0058] Specifically, the target discrete speech code is determined through the following steps:

[0059] Based on the target text and the corresponding prompt text and prompt voice code s of the style prompt voice. :t′ By reasoning, the decoded code s is obtained. t′:t′+t-1 , where t represents the decoding step size, and the length of the prompt speech encoding is t′+1;

[0060] If the current inference count is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. (i.e., the segment of speech code from position t′-w1 to position t′ in the speech code obtained by the text-to-discrete speech coding module, which contains w1 discrete speech codes) the probability value at the corresponding index, and reduced by a factor of σ, where n represents the continuous repetition threshold (its value is related to the speech quantizer used), w1 represents the window length, and σ (≥1) represents the penalty coefficient;

[0061] If the current inference count is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. (That is, the segment of speech encoding from position -w2-n to position -n in the speech encoding obtained by the text-to-discrete speech encoding module, which contains w2 discrete speech encodings) the probability value at the corresponding index, and reduced by a factor of σ, where w2 represents the window length;

[0062] Divide the current probability value (reduced by a factor of σ) by the temperature coefficient to obtain the updated probability distribution. Based on this updated probability distribution, decode the speech code s for the current decoding time using multinomial sampling. tAnd based on the speech code s t The inference and decoding steps are performed iteratively until the final encoding is obtained, thus obtaining the target discrete speech code.

[0063] It should be noted that, considering that current large-scale TTS systems may experience instability such as missing words, repeated readings, and infinite loops during the inference process due to random sampling strategies, this application designs a penalty-before-sampling strategy to optimize the speech synthesis process.

[0064] Taking the text-to-discrete speech coding module as a pre-trained GPT model as an example, refer to... Figure 3 The flowchart shown is a diagram of the penalty-then-sampling strategy. Figure 3 Line 1 describes the meaning of each parameter; lines 2-5 describe the penalty process, that is, if the current inference count is less than n, the probability distribution output by the GPT model at the current time step will be obtained. The probability value at the corresponding index is reduced by a factor of σ. This is understandable because, at the beginning of inference, the inference result is easily affected by the indefinite-length silence at the end of the prompt speech. This operation effectively reduces the interference on the inference result (i.e., the probability distribution). If the current number of inferences is greater than or equal to n, the probability distribution output by the GPT model at the current time step is obtained. The probability value at the corresponding index is then reduced by a factor of σ. It should be noted that the number of consecutive repetitions of most discrete speech codes is unlikely to exceed n. The value of n is related to the speech quantizer used. In practical applications, the value of n can be determined by observing and statistically analyzing the features extracted from audio (e.g., tens of thousands of hours of audio). The probability value at the corresponding index is reduced by a factor of σ, which can make the decoded speech code have the same distribution as the real speech code as possible; lines 6-7 describe the post-sampling process, that is, first divide the probability value obtained in line 3 or line 5 by the temperature coefficient, and then decode the speech code s at the current time through polynomial sampling. t Subsequent decoding is based on a loop from line 2 to line 7, which is repeated until the final code is decoded, thus obtaining the target discrete speech code.

[0065] Step S13: Using a pre-trained discrete speech code to waveform module, the target discrete speech code is converted into the target speech based on the timbre prompt speech.

[0066] The discrete speech coding to waveform module can be built based on an end-to-end VITS architecture.

[0067] In specific implementation, the discrete speech coding to waveform module converts the target discrete speech code based on the timbre-cue speech, that is, it converts the target discrete speech code into the speaker's voice waveform corresponding to the timbre-cue speech, thereby obtaining the target speech (i.e., synthesized speech) that conforms to the style features of the style-cue speech and the speaker features of the timbre-cue speech.

[0068] The technical solution adopted in this application addresses the instability issues caused by random sampling strategies, such as missing words, repeated readings, and infinite loops. Therefore, this application designs a penalty-before-sampling strategy to optimize the speech synthesis process. Specifically, when the current inference count is less than n, the obtained speech code... The probability value at the corresponding index is reduced by a factor of σ to decrease the interference of the inference result caused by the variable-length silence segment at the end of the prompt speech at the beginning of inference; when the current number of inferences is not less than n, the obtained speech code is... The probability value at the corresponding index is reduced by a factor of σ to ensure that the decoded speech code has the same distribution as the real speech code as possible; then, the speech code s at the current decoding time is decoded based on the temperature coefficient and polynomial sampling. t The subsequent reasoning and decoding follow the aforementioned steps in a loop until the final encoding is obtained, resulting in the target discrete speech code. Finally, the target speech (i.e., synthesized speech) is obtained based on the target discrete speech code. This reduces the occurrence of unstable phenomena such as missing words, repeated readings, and infinite loops, thereby significantly improving the stability of synthesized speech.

[0069] As one possible implementation, the text-to-discrete speech coding module is an autoregressive language model (such as the GPT model), which is trained through the following steps:

[0070] Multiple audio samples are obtained from internet data, and the transcribed text corresponding to each of the multiple audio samples is obtained;

[0071] The speech quantizer encodes the multiple audio samples into discrete speech codes, resulting in multiple speech code samples.

[0072] For each of the speech coding samples, a discrete speech code of no more than a set length is randomly selected from the front end of the speech coding sample as a prompt speech coding sample;

[0073] For each speech coding sample, the prompt speech coding sample and the transcribed text corresponding to the speech coding sample are padded to the set length using padding encoding. Then, the prompt speech coding sample and the transcribed text are processed through an embedding layer, and the processed prompt speech coding sample and the transcribed text are added together to obtain reference information.

[0074] For each speech coding sample, the remaining discrete speech coding in the speech coding sample is processed by the embedding layer, and the reference information and the processed remaining discrete speech coding are concatenated to obtain the training sample;

[0075] The autoregressive language model is trained using the training samples corresponding to each of the multiple audio samples.

[0076] In practice, tens of thousands of hours of low-quality internet data can be collected to build a training dataset, which includes multiple audio samples y = {y0, y1, y2, ..., y...}. T} and its corresponding transcribed text x = {x0, x1, x2, ..., x} L Each audio sample y is encoded into a discrete speech code using a speech tokenizer (ST), i.e., ST(y) = s, thus obtaining the corresponding speech code sample s.

[0077] Then, for each speech coding sample, a discrete speech code of no more than a set length (e.g., no more than 256 characters) is randomly selected from the front end of the speech coding sample as the prompt speech coding sample.

[0078] Considering that the transcribed text (which is character-level encoded and its length does not exceed the set length) is uninterrupted and continuous, to ensure the continuity between the prompt speech encoding sample and the subsequent discrete speech encoding, padding encoding is used to fill the prompt speech encoding sample and the transcribed text corresponding to the speech encoding sample to the set length. For example, the padding encoding can be concatenated on one side (e.g., the left side) of the prompt speech encoding sample and the transcribed text so that both the prompt speech encoding sample and the transcribed text reach the set length. Then, the prompt speech encoding sample and the transcribed text are processed through an embedding layer, and the processed prompt speech encoding sample and the transcribed text are added together to obtain reference information.

[0079] The remaining discrete speech codes in the speech coding sample are processed by the embedding layer, and the reference information and the processed remaining discrete speech codes are concatenated to obtain the training sample corresponding to the audio sample.

[0080] Finally, the training samples corresponding to each of the multiple audio samples obtained through the above steps are input into the autoregressive language model, for example, the multiple training samples are input into the GPT model through the position encoding layer for model training.

[0081] In this embodiment, considering the widespread success of the autoregressive language model paradigm for modeling discrete coding in the field of Natural Language Processing (NLP), this application adopts an autoregressive language model-based approach (such as the GPT model) to model the task of converting discrete text coding to discrete speech coding, in order to ensure the conversion effect. Furthermore, considering current speech synthesis methods, the text-to-discrete speech coding modeling stage mainly continues the VALL-E AR approach, that is, using a simple concatenation-style prompt design. Specifically, refer to... Figure 4 The diagram illustrates the training process of a VALL-E AR-like system. Current speech synthesis methods concatenate speech codes after text codes during training. During inference, they arbitrarily select a prompt audio of about 3 seconds, transcribe its text, concatenate it before the input text, and then concatenate the speech code of the prompt audio to allow the model to continue generating the speech code corresponding to the input text. However, this simple concatenation-style prompt design makes it difficult to clearly define the role of the prompt audio during training, resulting in the audio generated by the model during inference failing to maintain the speech rate, style, and emotion of the prompt audio.

[0082] To solve the above problems, refer to Figure 5 The diagram shows the training process of the text-to-discrete speech coding module. This application proposes a "double coherent addition" prompt design, which uses the above training samples for model training to clarify the role of the prompt audio. This allows the trained model to better maintain the speech rate, style, and emotion of the style prompt speech during the inference stage. Compared with the simple concatenation prompt design, the "double coherent addition" prompt design proposed in this application has lower model computational complexity, which is beneficial to improving model computational efficiency.

[0083] Optionally, the reference information is determined by the following formula:

[0084] c = Embedding([pad]||s 0:t +Embedding([pad]||x)

[0085] Where 'c' represents reference information, 'Embedding' represents the embedding layer, '[pad]' represents padding encoding, '||' represents the concatenation operation, and 's' represents the reference information. 0:t The text indicates the voice encoding sample (t ≤ the set length), and x represents the transcribed text.

[0086] Optionally, the training samples are determined using the following formula:

[0087] h = c ||Embedding(s) t+1: )

[0088] Where h represents the training samples, c represents the reference information, Embedding represents the embedding layer, || represents the concatenation operation, and s t+1: This represents the remaining discrete speech codes in the speech coding sample (t≤the set length), that is, the other discrete speech codes in the speech coding sample besides the extracted prompt speech coding sample.

[0089] Optionally, the total training loss of the autoregressive language model during training is determined by the following formula:

[0090]

[0091] Where L represents the total training loss, This represents the text loss (padding encoding can be ignored when calculating the text loss). The α and β represent speech loss, and the weights represent weights. In some implementations, α and β can be set to 0.2 and 0.8, respectively.

[0092] It should be noted that this application's embodiments train an autoregressive language model (such as an autoregressive GPT model, which uses text and audio prompts as conditions) to model the mapping relationship from text to discrete speech codes. Considering that only when the model can better understand the text content can it more naturally and stably map the text to the corresponding discrete speech codes, this application adds explicit modeling of the text; that is, it introduces an additional text loss during model training to construct a bidirectional loss function for text and speech to obtain the total training loss. In this way, the model can consider more of the semantic information contained in the text when referring to the audio prompts, thereby giving the synthesized speech a natural rhythm and realism.

[0093] Optionally, text loss Determined by the following formula:

[0094]

[0095] Among them, P θ (x t |h ＜t ) indicates that the autoregressive language model is trained on sample h. ＜t To learn transcribed text x as a condition t The target distribution, where D represents the set length (e.g., 256).

[0096] Optionally, speech loss Determined by the following formula:

[0097]

[0098] Among them, P θ (st |h ＜t ) indicates that the autoregressive language model is trained on sample h. ＜t To learn speech coding samples s t Target distribution, s t The length of the corresponding audio sample is T+1, and D represents the set length (e.g., 256).

[0099] As one possible implementation, the speech quantizer is a kmene model, which is trained through the following steps:

[0100] Intermediate representations of layer 15 were extracted from the XLSR-53 model;

[0101] The k-means model is trained to cluster the extracted intermediate representations to obtain discrete speech codes.

[0102] In this implementation, the extracted intermediate representations are clustered (i.e., cluster compression) by training a k-means model to obtain discrete speech codes, so as to ensure that the obtained discrete speech codes can retain more semantic information and less acoustic information (such as timbre).

[0103] It should be noted that, compared to related technologies that use multiple quantizers in a neural encoder to obtain multi-layer codes, and then model these multi-layer codes separately using an autoregressive and non-autoregressive architecture to achieve speech synthesis, this application constructs a speech quantizer based on a k-means model to quantize the original speech into a single-layer codebook (i.e., discrete speech codes). Then, a text-to-discrete speech coding module converts the input text into the corresponding discrete speech codes. Finally, a discrete speech coding-to-waveform module converts the discrete speech codes generated in the previous stage into high-quality speech. Thus, this application provides a single-layer codebook-based speech synthesis method, which can effectively improve training efficiency and inference speed.

[0104] Optionally, an end-to-end VITS architecture can be used to construct the discrete speech code-to-waveform module, specifically by replacing the feature extractor in the end-to-end VITS architecture with the speech quantizer constructed based on the aforementioned k-means model.

[0105] Optionally, considering that discrete speech coding can retain more semantic information and less timbre information due to the clustering operation of the k-means model, and also considering the need for lightweight deployment, the feature matching module and F0 module in the end-to-end VITS architecture can be further removed to obtain a discrete speech coding to waveform module. This discrete speech coding to waveform module uses the timbre-cue speech as the target audio, and the synthesized speech can still maintain a high timbre similarity with the timbre-cue speech.

[0106] As one possible implementation, the model training processes of the text-to-discrete speech coding module and the discrete speech coding-to-waveform module are independent of each other.

[0107] For specific implementation, see Figure 6 The training process diagram shown allows for the training of a text-to-discrete speech coding module using text and its corresponding discrete speech coding, and for the training of a discrete speech coding to waveform module using discrete speech coding and its corresponding speech, thus making the training processes of the two independent and accelerating the training efficiency of the base model.

[0108] Based on the above embodiments, this application achieves an improvement in the speech synthesis method, and the relevant improvements include at least the following:

[0109] A speech quantizer based on the kmeans model is constructed to quantize the original speech into a single codebook, thereby avoiding the problems of low training efficiency and slow inference speed caused by using multiple codebooks.

[0110] This paper proposes a two-stage model. In the first stage, the focus is on designing an autoregressive language model (such as the GPT model) to learn the ability to convert text to discrete speech codes, ensuring the model can accurately capture and understand semantic content. Specifically, in the training phase, this application proposes a "double coherent additive" cue design and a bidirectional loss function for text and speech, enabling the model not only to understand the semantics of the text but also to adjust the speech output according to the characteristics of different speakers, thereby achieving a more personalized and hyper-human speech synthesis effect. In the inference phase, this application proposes a penalty-before-sampling strategy, solving a series of instability problems such as repetition and missing words, significantly improving inference stability. In the second stage, a timbre conversion model (i.e., a discrete speech code to waveform module) is adopted, and speaker information (i.e., timbre-cued speech) is introduced to enable the model to learn to convert discrete speech codes into the voice waveforms of specific speakers.

[0111] Based on the above improvements, the speech generated by the speech synthesis method provided in this application can perfectly replicate the style of the style-cued speech and also conform to the speaker characteristics of the timbre-cued speech. It is worth mentioning that the speech synthesis method provided in this application has been successfully applied to the Xingchen Supernatural Speech Generation Model and launched on the Xingchen Huishuo platform; furthermore, in the ICAGC 2024 Highly Engaging and Credible Audio Generation Challenge, the speech synthesis method provided in this application scored 3.89 in audio quality, 3.83 in timbre similarity, and 3.85 in emotional expressiveness, with a MOS mean and standard deviation of 3.86 and 0.22 respectively, ranking first in all five evaluation indicators.

[0112] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of this application.

[0113] Secondly, Figure 7 This is a schematic diagram of a speech synthesis device according to an embodiment of this application. The device includes:

[0114] The first processing module 710 is used to encode the style prompt speech into discrete speech code through a pre-trained speech quantizer to obtain the prompt speech code;

[0115] The second processing module 720 is used to perform multiple inferences based on the target text and the prompt text and prompt speech code corresponding to the style prompt speech through a pre-trained text-to-discrete speech coding module to obtain the target discrete speech code corresponding to the target text.

[0116] The speech synthesis module 730 is used to convert the target discrete speech code into target speech using a pre-trained discrete speech code to waveform module, with timbre prompt speech as a condition;

[0117] The second processing module 720 is specifically used to perform the following steps:

[0118] Based on the target text and the corresponding prompt text and prompt voice code s of the style prompt voice. :t′ By reasoning, the decoded code s is obtained. t′:t′+t-1 Where t represents the decoding step size;

[0119] If the current inference count is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is reduced by a factor of σ, where n represents the consecutive repetition threshold, w1 represents the window length, and σ represents the penalty coefficient.

[0120] If the current inference count is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is then reduced by a factor of σ, where w2 represents the window length;

[0121] Divide the current probability value (reduced by a factor of σ) by the temperature coefficient to obtain the updated probability distribution. Based on this updated probability distribution, decode the speech code s for the current decoding time using multinomial sampling. t And based on the speech code s t The inference and decoding steps are performed iteratively until the final encoding is obtained, thus obtaining the target discrete speech code.

[0122] The technical solution adopted in this application addresses the instability issues caused by random sampling strategies, such as missing words, repeated readings, and infinite loops. Therefore, this application designs a penalty-before-sampling strategy to optimize the speech synthesis process. Specifically, when the current inference count is less than n, the obtained speech code... The probability value at the corresponding index is reduced by a factor of σ to decrease the interference of the inference result caused by the variable-length silence segment at the end of the prompt speech at the beginning of inference; when the current number of inferences is not less than n, the obtained speech code is... The probability value at the corresponding index is reduced by a factor of σ to ensure that the decoded speech code has the same distribution as the real speech code as possible; then, the speech code s at the current decoding time is decoded based on the temperature coefficient and polynomial sampling. t The subsequent reasoning and decoding follow the aforementioned steps in a loop until the final encoding is obtained, resulting in the target discrete speech code. Finally, the target speech (i.e., synthesized speech) is obtained based on the target discrete speech code. This reduces the occurrence of unstable phenomena such as missing words, repeated readings, and infinite loops, thereby significantly improving the stability of synthesized speech.

[0123] Optionally, the text-to-discrete speech coding module is an autoregressive language model, and the device further includes a first training module for performing the following steps to train the autoregressive language model:

[0124] Multiple audio samples are obtained from internet data, and the transcribed text corresponding to each of the multiple audio samples is obtained;

[0125] The speech quantizer encodes the multiple audio samples into discrete speech codes, resulting in multiple speech code samples.

[0126] For each of the speech coding samples, a discrete speech code of no more than a set length is randomly selected from the front end of the speech coding sample as a prompt speech coding sample;

[0127] For each speech coding sample, the prompt speech coding sample and the transcribed text corresponding to the speech coding sample are padded to the set length using padding encoding. Then, the prompt speech coding sample and the transcribed text are processed through an embedding layer, and the processed prompt speech coding sample and the transcribed text are added together to obtain reference information.

[0128] For each speech coding sample, the remaining discrete speech coding in the speech coding sample is processed by the embedding layer, and the reference information and the processed remaining discrete speech coding are concatenated to obtain the training sample;

[0129] The autoregressive language model is trained using the training samples corresponding to each of the multiple audio samples.

[0130] Optionally, the reference information is determined by the following formula:

[0131] c = Embedding([pad]||s 0:t +Embedding([pad]||x)

[0132] Where 'c' represents reference information, 'Embedding' represents the embedding layer, '[pad]' represents padding encoding, '||' represents the concatenation operation, and 's' represents the reference information. 0:t This indicates a voice-coded prompt sample, where x represents the transcribed text.

[0133] Optionally, the training samples are determined using the following formula:

[0134] h = c ||Embedding(s) t+1: )

[0135] Where h represents the training samples, c represents the reference information, Embedding represents the embedding layer, || represents the concatenation operation, and s t+1: This represents the remaining discrete speech codes in the speech coding sample.

[0136] Optionally, the total training loss of the autoregressive language model during training is determined by the following formula:

[0137]

[0138] Where L represents the total training loss, Indicates text loss, Let α represent speech loss, and β represent weights.

[0139] Optionally, text loss Determined by the following formula:

[0140]

[0141] Among them, P θ (x t |h ＜t ) indicates that the autoregressive language model is trained on sample h. ＜t To learn transcribed text x as a condition t The target distribution, where D represents the set length.

[0142] Optionally, speech loss L a Determined by the following formula:

[0143]

[0144] Among them, P θ (s t |h ＜t ) indicates that the autoregressive language model is trained on sample h. ＜t To learn speech coding samples s t Target distribution, s t The length of the corresponding audio sample is T+1, and D represents the set length.

[0145] Optionally, the speech quantizer is a kmean model, and the device further includes a second training module for performing the following steps to train the kmean model:

[0146] Intermediate representations of layer 15 were extracted from the XLSR-53 model;

[0147] The k-means model is trained to cluster the extracted intermediate representations to obtain discrete speech codes.

[0148] Optionally, the model training processes of the text-to-discrete speech coding module and the discrete speech coding-to-waveform module are independent of each other.

[0149] It should be noted that the device embodiments are similar to the method embodiments, so the description is relatively simple. For relevant details, please refer to the method embodiments.

[0150] This application also provides an electronic device, see embodiments thereof. Figure 8 , Figure 8 This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 8 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the speech synthesis method disclosed in the embodiments of this application.

[0151] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the speech synthesis method disclosed in this application.

[0152] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the speech synthesis method disclosed in this application.

[0153] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This application describes embodiments of methods, systems, devices, storage media, and program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0159] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0160] The above provides a detailed description of a speech synthesis method, apparatus, device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech synthesis method, characterized in that, The method includes: The style cue speech is encoded into discrete speech codes by using a pre-trained speech quantizer, thus obtaining the cue speech code. By using a pre-trained text-to-discrete speech coding module, multiple inferences are performed based on the target text and the prompt text and prompt speech coding corresponding to the style prompt speech to obtain the target discrete speech coding corresponding to the target text. The target discrete speech code is converted into target speech by a pre-trained discrete speech code-to-waveform module, based on the timbre-cue speech; The target discrete speech code is determined through the following steps: Based on the target text and the corresponding prompt text and prompt voice code s of the style prompt voice. :t′ By reasoning, the decoded code s is obtained. t′:t′+t-1 Where t represents the decoding step size; If the current inference count is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is reduced by a factor of σ, where n represents the consecutive repetition threshold, w1 represents the window length, and σ represents the penalty coefficient. If the current inference count is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is then reduced by a factor of σ, where w2 represents the window length; Divide the current probability value (reduced by a factor of σ) by the temperature coefficient to obtain the updated probability distribution. Based on this updated probability distribution, decode the speech code s for the current decoding time using multinomial sampling. t And based on the speech code s t The inference and decoding steps are performed iteratively until the final encoding is obtained, thus obtaining the target discrete speech code.

2. The method according to claim 1, characterized in that, The text-to-discrete speech coding module is an autoregressive language model, which is trained through the following steps: Multiple audio samples are obtained from internet data, and the transcribed text corresponding to each of the multiple audio samples is obtained; The speech quantizer encodes the multiple audio samples into discrete speech codes, resulting in multiple speech code samples. For each of the speech coding samples, a discrete speech code of no more than a set length is randomly selected from the front end of the speech coding sample as a prompt speech coding sample; For each speech coding sample, the prompt speech coding sample and the transcribed text corresponding to the speech coding sample are padded to the set length using padding encoding. Then, the prompt speech coding sample and the transcribed text are processed through an embedding layer, and the processed prompt speech coding sample and the transcribed text are added together to obtain reference information. For each speech coding sample, the remaining discrete speech coding in the speech coding sample is processed by the embedding layer, and the reference information and the processed remaining discrete speech coding are concatenated to obtain the training sample; The autoregressive language model is trained using the training samples corresponding to each of the multiple audio samples.

3. The method according to claim 2, characterized in that, The reference information is determined by the following formula: c=Embedding([pad]||s 0:t )+Embedding([pad]||x) Where 'c' represents reference information, 'Embedding' represents the embedding layer, '[pad]' represents padding encoding, '||' represents the concatenation operation, and 's' represents the reference information. 0:t This indicates a voice-coded prompt sample, where x represents the transcribed text.

4. The method according to claim 2, characterized in that, The training samples are determined using the following formula: h=c||Embedding(s t+1: ) Where h represents the training samples, c represents the reference information, Embedding represents the embedding layer, || represents the concatenation operation, and s t+1: This represents the remaining discrete speech codes in the speech coding sample.

5. The method according to claim 2, characterized in that, The total training loss of the autoregressive language model during training is determined by the following formula: Where L represents the total training loss, Indicates text loss, Let α represent speech loss, and β represent weights.

6. The method according to claim 5, characterized in that, Text loss Determined by the following formula: Among them, P θ (x t |h ＜t ) indicates that the autoregressive language model is trained on sample h. ＜t To learn transcribed text x as a condition t The target distribution, where D represents the set length.

7. The method according to claim 5, characterized in that, Voice loss Determined by the following formula: Among them, P θ (s t |h ＜t ) indicates that the autoregressive language model is trained on sample h. ＜t To learn speech coding samples s t Target distribution, s t The length of the corresponding audio sample is T+1, and D represents the set length.

8. The method according to any one of claims 1-7, characterized in that, The speech quantizer is a kmene model, which is trained through the following steps: Intermediate representations of layer 15 were extracted from the XLSR-53 model; The k-means model is trained to cluster the extracted intermediate representations to obtain discrete speech codes.

9. The method according to any one of claims 1-7, characterized in that, The model training processes of the text-to-discrete speech coding module and the discrete speech coding-to-waveform module are independent of each other.

10. A speech synthesis device, characterized in that, The device comprises: The first processing module is used to encode the style prompt speech into discrete speech codes using a pre-trained speech quantizer, thereby obtaining the prompt speech code; The second processing module is used to perform multiple inferences based on the target text and the prompt text and prompt speech code corresponding to the style prompt speech through a pre-trained text-to-discrete speech coding module to obtain the target discrete speech code corresponding to the target text. The speech synthesis module is used to convert the target discrete speech code into target speech by using a pre-trained discrete speech code to waveform module, with timbre prompt speech as a condition; Specifically, the second processing module is used to perform the following steps: Based on the target text and the corresponding prompt text and prompt voice code s of the style prompt voice. :t′ By reasoning, the decoded code s is obtained. t′:t′+t-1 Where t represents the decoding step size; If the current inference count is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is reduced by a factor of σ, where n represents the consecutive repetition threshold, w1 represents the window length, and σ represents the penalty coefficient. If the current inference count is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech coding module at the current decoding time. The probability value at the corresponding index is then reduced by a factor of σ, where w2 represents the window length; Divide the current probability value (reduced by a factor of σ) by the temperature coefficient to obtain the updated probability distribution. Based on this updated probability distribution, decode the speech code s for the current decoding time using multinomial sampling. t And based on the speech code s t The inference and decoding steps are performed iteratively until the final encoding is obtained, thus obtaining the target discrete speech code.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the speech synthesis method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the speech synthesis method as described in any one of claims 1 to 9.

Images