Speech synthesis method, device, equipment and medium

Abstract

The invention provides a voice synthesis method and device, equipment and a medium, and belongs to the technical field of computers, and the method comprises the steps: encoding a style prompt voice into a discrete voice code through a pre-trained voice quantizer, and obtaining a prompt voice code; through a pre-trained text-to-discrete speech coding module, based on a punishment-first and sampling-second strategy, according to a target text and a prompt text and a prompt speech code corresponding to the style prompt speech, performing multiple reasoning to obtain a target discrete speech code corresponding to the target text; and through a pre-trained discrete voice code-to-waveform module, converting the target discrete voice code into target voice by taking tone prompt voice as a condition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech synthesis method, apparatus, device and medium. Background Art

[0002] With the continuous development of deep learning and the popularization of intelligent devices, text-to-speech (TTS) technology has been widely used in many fields such as intelligent assistants, navigation systems, and educational tools.

[0003] In the related art, a neural encoder model is usually first used to quantize the speech waveform into discrete speech codes, and then an autoregressive language model (LM) is used for encoding and decoding modeling, and a random sampling strategy is used for inference to achieve speech synthesis.

[0004] However, the random sampling strategy may lead to unstable phenomena such as missing words, repeated reading, and infinite loops, resulting in poor stability of the synthesized speech. Summary of the invention

[0005] In view of the above problems, the embodiments of the present application provide a speech synthesis method, apparatus, device and medium to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a speech synthesis method is provided, the method comprising:

[0007] The style cue speech is encoded into discrete speech codes by a pre-trained speech quantizer to obtain the cue speech code;

[0008] Through a pre-trained text-to-discrete speech coding module, multiple inferences are performed based on the target text and the prompt text and prompt speech coding corresponding to the style prompt speech to obtain the target discrete speech coding corresponding to the target text;

[0009] The target discrete speech code is converted into the target speech by a pre-trained discrete speech code to waveform module with the timbre prompt speech as a condition;

[0010] The target discrete speech coding is determined by the following steps:

[0011] According to the target text and the prompt text and prompt voice code corresponding to the style prompt voice :t′ Perform inference and obtain the decoded code s t′:t′+t-1 , where t represents the decoding step size;

[0012] When the current number of inferences is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where n represents the continuous repetition threshold, w1 represents the window length, and σ represents the penalty coefficient;

[0013] When the current number of inferences is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where w2 represents the window length;

[0014] The probability value obtained after being reduced by σ times is divided by the temperature coefficient to obtain an updated probability distribution, and the speech code s at the current decoding time is decoded by polynomial sampling according to the updated probability distribution. t , and based on the speech coding s t The steps related to reasoning and decoding are iteratively performed until the target discrete speech code is obtained when the end code is decoded.

[0015] According to a second aspect of the embodiments of the present application, a speech synthesis device is provided, the device comprising:

[0016] A first processing module is used to encode the style cue speech into discrete speech codes through a pre-trained speech quantizer to obtain the cue speech code;

[0017] The second processing module is used to perform multiple inferences according to the target text and the prompt text and prompt voice code corresponding to the style prompt voice through a pre-trained text-to-discrete voice code module to obtain the target discrete voice code corresponding to the target text;

[0018] A speech synthesis module, used for converting the target discrete speech code into a target speech with the timbre prompt speech as a condition through a pre-trained discrete speech code to waveform module;

[0019] The second processing module is specifically configured to perform the following steps:

[0020] According to the target text and the prompt text and prompt voice code corresponding to the style prompt voice :t′ Perform inference and obtain the decoded code s t′:t′+t-1 , where t represents the decoding step size;

[0021] When the current number of inferences is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where n represents the continuous repetition threshold, w1 represents the window length, and σ represents the penalty coefficient;

[0022] When the current number of inferences is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where w2 represents the window length;

[0023] The probability value obtained after being reduced by σ times is divided by the temperature coefficient to obtain an updated probability distribution, and the speech code s at the current decoding time is decoded by polynomial sampling according to the updated probability distribution. t , and based on the speech coding s t The steps related to reasoning and decoding are iteratively performed until the target discrete speech code is obtained when the end code is decoded.

[0024] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the speech synthesis method described in the first aspect are implemented.

[0025] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the speech synthesis method described in the first aspect are implemented.

[0026] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the speech synthesis method as described in the first aspect.

[0027] The embodiments of the present application include the following advantages: Considering that the random sampling strategy may cause unstable phenomena such as missing words, repeated reading and infinite loops, the present application designs a penalty-first-then-sampling strategy to optimize the speech synthesis process. Specifically, when the current number of inferences is less than n, the acquired speech code is The probability value at the corresponding index is reduced by σ times to reduce the interference of the indefinite length of silence segment at the end of the prompt speech on the reasoning result at the beginning of the reasoning; when the current number of reasoning is not less than n, the obtained speech code The probability value at the corresponding index is reduced by σ times to ensure that the decoded speech code has the same distribution as the real speech code as much as possible; then the speech code s at the current decoding time is decoded based on the temperature coefficient and polynomial sampling. tThe subsequent reasoning and decoding are repeated in accordance with the aforementioned steps until the target discrete speech code is obtained when the end code is decoded, and finally the target speech (i.e., synthesized speech) is obtained according to the target discrete speech code. In this way, the occurrence of unstable phenomena such as missing words, repeated reading, and infinite loops can be reduced, thereby significantly improving the stability of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0029] Figure 1 is an implementation flow chart of a speech synthesis method in an embodiment of the present application;

[0030] Figure 2 It is a schematic diagram of a reasoning process in an embodiment of the present application;

[0031] Figure 3 This is a flowchart of a penalty-first-then-sampling strategy in an embodiment of the present application;

[0032] Figure 4 It is a VALL-EAR-like training process diagram in the related technology;

[0033] Figure 5 It is a schematic diagram of a training process of a text-to-discrete speech coding module in an embodiment of the present application;

[0034] Figure 6 is a schematic diagram of a training process in an embodiment of the present application;

[0035] Figure 7 is a structural schematic diagram of a speech synthesis device in an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0037] To facilitate understanding of the technical solutions provided by the present application, the main technical concepts involved in the embodiments of the present application are briefly described below.

[0038] Text-to-speech (TTS): aims to convert input text into natural and fluent speech output.

[0039] Generative Pre-trained Transformer (GPT) model: is a deep learning model based on the autoregressive language model. The core idea of ​​the autoregressive language model is to use the already generated words to predict the probability of the next word. In the GPT model, this autoregressive feature is reflected in the fact that the model will generate the next word based on the given context until a certain length of text is generated or a specific end symbol is encountered.

[0040] VITS: is an end-to-end text-to-speech architecture that uses variational inference coupled with a normalization process and adversarial training to produce more natural audio.

[0041] Embedding layer: It is a special layer in a neural network that is used to map discrete input data (such as words, characters, or tokens) into a continuous vector space.

[0042] XLSR-53 model: It is a pre-trained model for 53 languages ​​open sourced by Facebook.

[0043] Mean Opinion Score (MOS): is a measure of people’s average opinion score about a particular service, product, or experience.

[0044] In recent years, with the continuous development of deep learning and the popularization of intelligent devices, TTS technology has been widely used in many fields. For example: intelligent assistants (such as Siri and Xiao Ai) use TTS technology to provide a natural interactive experience, allowing users to easily obtain information and complete tasks through voice commands; navigation systems help drivers reach their destinations safely through voice prompts generated based on TTS technology, reducing dependence on screens; educational tools use TTS technology to provide convenience for the visually impaired and those with dyslexia, allowing them to acquire knowledge more easily. In addition, the emergence of the GPT-4O interaction mode has greatly improved the human-computer interaction experience, and the natural fluency of synthetic speech as a direct entrance to human-computer interaction directly determines the quality of the user experience.

[0045] In the related art, limited by the constraints of training data and training paradigm, traditional TTS systems usually use mel-spectrograms as intermediate representations and cascade modules such as prosody prediction, grapheme-to-phoneme, acoustic model, and vocoder. Such systems need to be trained with high-quality studio data, but recording high-quality data is very limited and expensive, and the speech synthesized by such systems is difficult to reach the level of humans. The main reasons are as follows:

[0046] On the one hand, spontaneous human speech contains not only language information but also a lot of subtle emotional paralinguistic information, while studio-recorded speech is usually read or performed according to a pre-set text, lacking diversity;

[0047] On the other hand, TTS systems trained with limited data generally have poor generalization capabilities and basically lack zero-sample synthesis capabilities. That is, for speakers that have never been seen, the naturalness and similarity of the synthesized speech generated by the TTS system will be severely reduced.

[0048] In contrast, large-scale TTS systems support training using tens of thousands of hours of data captured from the Internet, which enables them to have better zero-sample synthesis capabilities.

[0049] At present, speech synthesis methods based on large-scale TTS systems usually use a neural encoder model to quantize speech waveforms into discrete speech codes, then use an autoregressive language model for codec modeling, and use a random sampling strategy for inference to achieve speech synthesis. For example, the speech synthesis method based on VALL-E uses an autoregressive (AR) language model to encode coarse-grained codes, and a non-autoregressive (NAR) language model to encode the remaining fine-grained codes, and uses a random sampling strategy for inference.

[0050] However, since the random sampling strategy may cause unstable phenomena such as missing words, repeated reading, and infinite loops, the stability of the synthesized speech obtained by the current speech synthesis method based on large-scale TTS systems is relatively poor.

[0051] In response to the problems existing in the above-mentioned related technologies, the embodiments of the present application propose a speech synthesis method, device, equipment and medium, which optimizes the speech synthesis process based on the strategy of first punishing and then sampling, can reduce the occurrence of unstable phenomena such as missing words, repeated reading and infinite loops, thereby significantly improving the stability of the synthesized speech.

[0052] In the following, in combination with the accompanying drawings, a speech synthesis method, apparatus, device and medium provided in the embodiments of the present application are described in detail through some embodiments and their application scenarios.

[0053] First, refer to Figure 1 FIG. 1 is a flowchart of a speech synthesis method according to an embodiment of the present invention, wherein the method comprises the following steps:

[0054] Step S11: encoding the style cue speech into discrete speech codes through a pre-trained speech quantizer to obtain the cue speech codes.

[0055] The speech quantizer can be constructed based on a neural encoder model or a kmeans model.

[0056] In specific implementation, refer to Figure 2 The reasoning flow diagram shown in the figure uses a speech quantizer to encode the style cue speech into a discrete speech code, and determines the obtained discrete speech code as the cue speech code. Then, the cue speech code, the target text (i.e., the input text), and the cue text (i.e., the transcribed text) corresponding to the style cue speech are input into the text-to-discrete speech code module for the next step of processing.

[0057] Step S12: a pre-trained text-to-discrete speech coding module is used to perform multiple inferences based on the target text and the prompt text and prompt speech coding corresponding to the style prompt speech to obtain the target discrete speech coding corresponding to the target text.

[0058] Specifically, the target discrete speech coding is determined by the following steps:

[0059] According to the target text and the prompt text and prompt voice code corresponding to the style prompt voice :t′ Perform inference and obtain the decoded code s t′:t′+t-1 , where t represents the decoding step length, indicating that the length of the speech coding is t′+1;

[0060] When the current number of inferences is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. (i.e., the probability value at the corresponding index of the speech code from position t′-w1 to position t′ in the speech code obtained by the text-to-discrete speech coding module, which contains w1 discrete speech codes) and reduces it by σ times, where n represents the continuous repetition threshold (its value is related to the speech quantizer used), w1 represents the window length, and σ (≥1) represents the penalty coefficient;

[0061] When the current number of inferences is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. (i.e., a section of speech coding corresponding to the position -w2-n to the position -n in the speech coding obtained by the text-to-discrete speech coding module, which contains w2 discrete speech codings) the probability value at the corresponding index, and reduce it by σ times, where w2 represents the window length;

[0062] The probability value obtained after being reduced by σ times is divided by the temperature coefficient to obtain an updated probability distribution, and the speech code s at the current decoding time is decoded by polynomial sampling according to the updated probability distribution. t, and based on the speech coding s t The steps related to reasoning and decoding are iteratively performed until the target discrete speech code is obtained when the end code is decoded.

[0063] It should be noted that, considering that the current large-scale TTS system may have unstable phenomena such as missing words, repeated reading, and infinite loops due to random sampling strategies during the reasoning process, this application designs a penalty-first-then-sampling strategy to optimize the speech synthesis process.

[0064] Take the text-to-discrete speech encoding module as a pre-trained GPT model as an example, refer to Figure 3 The flowchart of the punishment-before-sampling strategy is shown in Figure 1. Figure 3 The first row in the middle describes the meaning of each parameter; rows 2-5 describe the process of first punishment, that is, if the current number of inferences is less than n, it will be obtained from the probability distribution output by the GPT model at the current moment. The probability value at the corresponding index is obtained by reducing it by σ times. It can be understood that since the inference result is easily disturbed by the indefinite length of silence at the end of the prompt speech at the beginning of the inference, this operation can effectively reduce the interference of the inference result (that is, the probability distribution); if the current number of inferences is greater than or equal to n, it will be obtained from the probability distribution output by the GPT model at the current moment The probability value at the corresponding index is reduced by σ times. It should be noted that the probability of most discrete speech codes being repeated continuously will not exceed n times. The value of n is related to the speech quantizer used. In practical applications, the value of n can be determined by observing and counting the features extracted from the audio (such as tens of thousands of hours of audio). The probability value at the corresponding index is reduced by σ times, which can make the decoded speech code have the same distribution as the real speech code as much as possible; Lines 6-7 describe the post-sampling process, that is, first divide the probability value obtained in line 3 or line 5 by the temperature coefficient, and then decode the speech code s at the current moment through polynomial sampling. t ; Subsequent decoding is repeated based on rows 2 to 7 in sequence until the end code is decoded to obtain the target discrete speech code.

[0065] Step S13: using a pre-trained discrete speech code to waveform module, the target discrete speech code is converted into a target speech with the timbre prompt speech as a condition.

[0066] The discrete speech coding to waveform module may be constructed based on an end-to-end VITS architecture.

[0067] In a specific implementation, the discrete speech code to waveform module converts the target discrete speech code based on the timbre prompt speech, that is, converts the target discrete speech code into the sound waveform of the speaker corresponding to the timbre prompt speech, thereby obtaining the target speech (i.e., synthesized speech) that conforms to the style characteristics of the style prompt speech and the speaker characteristics of the timbre prompt speech.

[0068] Using the technical solution of the embodiment of the present application, considering that the random sampling strategy may cause unstable phenomena such as missing words, repeated reading and infinite loops, the present application designs a penalty-first-then-sampling strategy to optimize the speech synthesis process. Specifically, when the current number of inferences is less than n, the acquired speech code is The probability value at the corresponding index is reduced by σ times to reduce the interference of the indefinite length of silence segment at the end of the prompt speech on the reasoning result at the beginning of the reasoning; when the current number of reasoning is not less than n, the obtained speech code The probability value at the corresponding index is reduced by σ times to ensure that the decoded speech code has the same distribution as the real speech code as much as possible; then the speech code s at the current decoding time is decoded based on the temperature coefficient and polynomial sampling. t The subsequent reasoning and decoding are repeated in accordance with the aforementioned steps until the target discrete speech code is obtained when the end code is decoded, and finally the target speech (i.e., synthesized speech) is obtained according to the target discrete speech code. In this way, the occurrence of unstable phenomena such as missing words, repeated reading, and infinite loops can be reduced, thereby significantly improving the stability of the synthesized speech.

[0069] As a possible implementation, the text-to-discrete speech encoding module is an autoregressive language model (such as a GPT model), and the autoregressive language model is trained by the following steps:

[0070] Acquire multiple audio samples from Internet data, and obtain transcription texts corresponding to each of the multiple audio samples;

[0071] Encoding the plurality of audio samples into discrete speech codes by the speech quantizer to obtain a plurality of speech code samples;

[0072] For each of the speech coding samples, randomly selecting a discrete speech coding not exceeding a set length from the front end of the speech coding sample as a prompt speech coding sample;

[0073] For each of the speech coding samples, padding the prompt speech coding sample and the transcribed text corresponding to the speech coding sample to the set length using padding coding, processing the prompt speech coding sample and the transcribed text through an embedding layer, and adding the processed prompt speech coding sample and the transcribed text to obtain reference information;

[0074] For each of the speech coding samples, the remaining discrete speech coding in the speech coding sample is processed by an embedding layer, and the reference information and the processed remaining discrete speech coding are concatenated to obtain a training sample;

[0075] The autoregressive language model is trained using training samples corresponding to each of the multiple audio samples.

[0076] In a specific implementation, tens of thousands of hours of low-quality Internet data can be collected to establish a training data set, wherein the training data set includes multiple audio samples y = {y0, y1, y2, ..., y T} and its corresponding transcription x={x0,x1,x2,…,x L}. Use a speech tokenizer (ST) to encode each audio sample y into a discrete speech code, that is, ST(y) = s, thereby obtaining the corresponding speech code sample s.

[0077] Then, for each speech coding sample, a discrete speech coding not exceeding a set length (for example, a length not exceeding 256) is randomly selected from the front end of the speech coding sample as a prompt speech coding sample.

[0078] Considering that the transcribed text (which is a word-level code and has a length not exceeding the set length) is uninterrupted and continuous, in order to ensure the continuity between the prompt voice coding sample and the subsequent discrete voice coding, the prompt voice coding sample and the transcribed text corresponding to the voice coding sample are filled to the set length respectively using a padding code. For example, the padding code can be spliced ​​on one side (such as the left side) of the prompt voice coding sample and the transcribed text so that the prompt voice coding sample and the transcribed text both reach the set length. The prompt voice coding sample and the transcribed text are then processed by an embedding layer, and the processed prompt voice coding sample and the transcribed text are added to obtain reference information.

[0079] The remaining discrete speech codes in the speech coding sample are processed through an embedding layer, and the reference information and the processed remaining discrete speech codes are concatenated to obtain a training sample corresponding to the audio sample.

[0080] Finally, the training samples corresponding to the multiple audio samples obtained through the above steps are respectively input into the autoregressive language model, for example, the multiple training samples are input into the GPT model through the position encoding layer to perform model training.

[0081] In this implementation, considering that the paradigm of modeling discrete coding with autoregressive language model has achieved widespread success in the field of natural language processing (NLP), this application adopts a method based on autoregressive language model (such as GPT model) to model the task of converting discrete text coding to discrete speech coding to ensure the conversion effect. And considering that the current speech synthesis method mainly continues the VALL-E AR approach in the modeling stage of text to discrete speech coding, that is, adopts a simple splicing prompt design. Specifically, refer to Figure 4 As shown in the figure, the current speech synthesis method will splice the speech code after the text code during training, and arbitrarily select a prompt audio of about 3 seconds during inference, splice its transcribed text before the input text, and then splice the speech code of the prompt audio to allow the model to continue to generate the speech code corresponding to the input text. However, this simple splicing prompt design makes it difficult to clearly define the role of the prompt audio during training, resulting in the audio generated by the model during inference being unable to well continue the speed, style and emotion of the prompt audio.

[0082] To solve the above problems, refer to Figure 5 As shown in the training process diagram of the text-to-discrete speech coding module, the present application proposes a "double-coherent additive" prompt design, that is, the model is trained using the above-mentioned training samples to clarify the role of the prompt audio, so that the trained model can better continue the speaking speed, style and emotion of the style prompt voice in the reasoning stage; and compared with the simple splicing prompt design, the "double-coherent additive" prompt design proposed in the present application has a lower model calculation complexity, which is conducive to improving the model calculation efficiency.

[0083] Optionally, the reference information is determined by the following formula:

[0084] c=Embedding([pad]||s 0:t )+Embedding([pad]||x)

[0085] Among them, c represents reference information, Embedding represents embedding layer, [pad] represents padding encoding, || represents concatenation operation, s 0:t represents the prompt speech coding sample (t≤the set length), and x represents the transcribed text.

[0086] Optionally, the training sample is determined by the following formula:

[0087] h=c||Embedding(s t+1: )

[0088] Among them, h represents the training sample, c represents the reference information, Embedding represents the embedding layer, || represents the concatenation operation, and s t+1: It represents the remaining discrete speech codes in the speech coding sample (t≤the set length), that is, it represents other discrete speech codes in the speech coding sample except the extracted prompt speech coding sample.

[0089] Optionally, the total training loss of the autoregressive language model during the training process is determined by the following formula:

[0090]

[0091] Among them, L represents the total training loss, represents the text loss (padding encoding can be ignored when calculating text loss), represents speech loss, α and β represent weights. In some implementations, α and β may be set to 0.2 and 0.8, respectively.

[0092] It should be noted that the embodiment of the present application trains an autoregressive language model (such as an autoregressive GPT model, which uses text and prompt audio as conditions) to model the mapping relationship between text and discrete speech codes. Considering that only when the model can better understand the text content can the text be mapped to the corresponding discrete speech code more naturally and stably, the present application adds additional explicit modeling of the text, that is, additional text loss is introduced during model training to form a bidirectional loss function of text and speech to obtain the total training loss. In this way, the model can take into account the semantic information contained in the text when referring to the prompt audio, thereby giving the synthesized speech a natural rhythm and sense of reality.

[0093] Optionally, text loss Determined by the following formula:

[0094]

[0095] Among them, P θ (x t |h ＜t ) represents the autoregressive language model with training sample h ＜t Conditional on learning to transcribe text x t , where D represents the set length (eg, 256).

[0096] Optionally, speech loss Determined by the following formula:

[0097]

[0098] Among them, P θ (st |h ＜t ) represents the autoregressive language model with training sample h ＜t Conditional on learning speech coding samples s t The target distribution, s t The length of the corresponding audio sample is T+1, and D represents the set length (eg, 256).

[0099] As a possible implementation, the speech quantizer is a kmeans model, and the kmeans model is trained by the following steps:

[0100] Extract the intermediate representation of the 15th layer for the XLSR-53 model;

[0101] The kmeans model is trained to cluster the extracted intermediate representations to obtain discrete speech codes.

[0102] In this embodiment, the extracted intermediate representations are clustered (ie, clustered and compressed) by training a kmeans model to obtain discrete speech codes, so as to ensure that the obtained discrete speech codes can retain more semantic information and less acoustic information (such as timbre, etc.).

[0103] It should be noted that compared with the related art in which a neural encoder uses multiple quantizers to obtain multi-layer codes, and then the autoregressive plus non-autoregressive architecture is used to model the multi-layer codes to achieve speech synthesis, this application builds a speech quantizer based on the kmeans model to quantize the original speech into a layer of codebook (i.e., discrete speech code), and then the text-to-discrete speech code module converts the input text into the corresponding discrete speech code, and finally the discrete speech code-to-waveform module converts the discrete speech code generated in the previous stage into high-quality speech. In this way, this application provides a speech synthesis method based on a layer of codebook, which can effectively improve the training efficiency and reasoning speed.

[0104] Optionally, an end-to-end VITS architecture is used to construct a discrete speech coding to waveform module, specifically by replacing the feature extractor in the end-to-end VITS architecture with the speech quantizer constructed based on the kmeans model.

[0105] Optionally, considering that discrete speech coding can retain more semantic information and less timbre information due to the clustering operation of the kmeans model, and considering the lightweight deployment requirements, the feature matching module and the F0 module in the end-to-end VITS architecture can be further removed to obtain a discrete speech coding to waveform module. The discrete speech coding to waveform module takes the timbre cue speech as the target audio, and the synthesized speech can still maintain a high timbre similarity with the timbre cue speech.

[0106] As a possible implementation, the model training processes of the text-to-discrete speech coding module and the discrete speech coding-to-waveform module are independent of each other.

[0107] In the specific implementation, see Figure 6 As shown in the training process diagram, the text-to-discrete speech coding module can be trained using text and its corresponding discrete speech coding, and the discrete speech coding-to-waveform module can be trained using discrete speech coding and its corresponding speech, so that the training processes of the two are independent, thereby accelerating the training efficiency of the basic model.

[0108] Based on the above embodiments, the present application achieves an improvement in the speech synthesis method, and the related improvements include at least:

[0109] A speech quantizer is constructed based on the kmeans model to quantize the original speech into a layer of codebook, thereby avoiding the problems of low training efficiency and slow inference speed caused by using multi-layer codebooks.

[0110] A two-stage model is proposed. In the first stage, the focus is on designing an autoregressive language model (such as the GPT model) to learn the conversion capability from text to discrete speech coding to ensure that the model can accurately capture and understand the semantic content; specifically, in the training stage, this application proposes a "bi-coherent additive" prompt design and designs a bidirectional loss function for text and speech, so that the model can not only understand the semantics of the text, but also adjust the speech output according to the characteristics of different speakers, thereby achieving a more personalized and super-human speech synthesis effect; in the reasoning stage, this application proposes a strategy of first punishing and then sampling, which solves a series of instability problems such as repetition and missing words, and significantly improves the reasoning stability. In the second stage, a timbre conversion model (i.e., a discrete speech coding to waveform module) is adopted and speaker information (i.e., timbre prompt speech) is introduced to enable the model to learn to convert discrete speech coding into a sound waveform of a specific speaker.

[0111] Based on the above improvements, the speech generated by the speech synthesis method provided by this application can perfectly reproduce the style of the style prompt speech, and can also meet the speaker characteristics of the timbre prompt speech. It is worth mentioning that the speech synthesis method provided by this application has been successfully applied to the Xingchen Supernatural Speech Generation Model and launched on the Xingchen Huishuo platform; and in the ICAGC 2024 High Infectiousness and Trustworthy Audio Generation Challenge, the speech synthesis method provided by this application scored 3.89 in audio quality, 3.83 in timbre similarity, and 3.85 in emotional expression. The MOS average and standard deviation scores were 3.86 and 0.22 respectively, ranking first in all five evaluation indicators.

[0112] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0113] Second, Figure 7 : is a structural schematic diagram of a speech synthesis device according to an embodiment of the present application, the device comprising:

[0114] The first processing module 710 is used to encode the style cue speech into discrete speech codes through a pre-trained speech quantizer to obtain the cue speech code;

[0115] The second processing module 720 is used to perform multiple inferences according to the target text and the prompt text and prompt voice code corresponding to the style prompt voice through a pre-trained text-to-discrete voice code module to obtain the target discrete voice code corresponding to the target text;

[0116] The speech synthesis module 730 is used to convert the target discrete speech code into the target speech with the timbre prompt speech as a condition through the pre-trained discrete speech code to waveform module;

[0117] The second processing module 720 is specifically configured to perform the following steps:

[0118] According to the target text and the prompt text and prompt voice code corresponding to the style prompt voice :t′ Perform inference and obtain the decoded code s t′:t′+t-1 , where t represents the decoding step size;

[0119] When the current number of inferences is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where n represents the continuous repetition threshold, w1 represents the window length, and σ represents the penalty coefficient;

[0120] When the current number of inferences is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where w2 represents the window length;

[0121] The probability value obtained after being reduced by σ times is divided by the temperature coefficient to obtain an updated probability distribution, and the speech code s at the current decoding time is decoded by polynomial sampling according to the updated probability distribution. t , and based on the speech coding s t The steps related to reasoning and decoding are iteratively performed until the target discrete speech code is obtained when the end code is decoded.

[0122] Using the technical solution of the embodiment of the present application, considering that the random sampling strategy may cause unstable phenomena such as missing words, repeated reading and infinite loops, the present application designs a penalty-first-then-sampling strategy to optimize the speech synthesis process. Specifically, when the current number of inferences is less than n, the acquired speech code is The probability value at the corresponding index is reduced by σ times to reduce the interference of the indefinite length of silence segment at the end of the prompt speech on the reasoning result at the beginning of the reasoning; when the current number of reasoning is not less than n, the obtained speech code The probability value at the corresponding index is reduced by σ times to ensure that the decoded speech code has the same distribution as the real speech code as much as possible; then the speech code s at the current decoding time is decoded based on the temperature coefficient and polynomial sampling. t The subsequent reasoning and decoding are repeated in accordance with the aforementioned steps until the target discrete speech code is obtained when the end code is decoded, and finally the target speech (i.e., synthesized speech) is obtained according to the target discrete speech code. In this way, the occurrence of unstable phenomena such as missing words, repeated reading, and infinite loops can be reduced, thereby significantly improving the stability of the synthesized speech.

[0123] Optionally, the text-to-discrete speech encoding module is an autoregressive language model, and the device further includes a first training module, configured to perform the following steps to train the autoregressive language model:

[0124] Acquire multiple audio samples from Internet data, and obtain transcription texts corresponding to each of the multiple audio samples;

[0125] Encoding the plurality of audio samples into discrete speech codes by the speech quantizer to obtain a plurality of speech code samples;

[0126] For each of the speech coding samples, randomly selecting a discrete speech coding not exceeding a set length from the front end of the speech coding sample as a prompt speech coding sample;

[0127] For each of the speech coding samples, padding the prompt speech coding sample and the transcribed text corresponding to the speech coding sample to the set length using padding coding, processing the prompt speech coding sample and the transcribed text through an embedding layer, and adding the processed prompt speech coding sample and the transcribed text to obtain reference information;

[0128] For each of the speech coding samples, the remaining discrete speech coding in the speech coding sample is processed by an embedding layer, and the reference information and the processed remaining discrete speech coding are concatenated to obtain a training sample;

[0129] The autoregressive language model is trained using training samples corresponding to each of the multiple audio samples.

[0130] Optionally, the reference information is determined by the following formula:

[0131] c=Embedding([pad]||s 0:t )+Embedding([pad]||x)

[0132] Among them, c represents reference information, Embedding represents embedding layer, [pad] represents padding encoding, || represents concatenation operation, s 0:t represents the prompt speech encoding sample, and x represents the transcribed text.

[0133] Optionally, the training sample is determined by the following formula:

[0134] h=c||Embedding(s t+1: )

[0135] Among them, h represents the training sample, c represents the reference information, Embedding represents the embedding layer, || represents the concatenation operation, and s t+1: Represents the remaining discrete speech codes in the speech coding samples.

[0136] Optionally, the total training loss of the autoregressive language model during the training process is determined by the following formula:

[0137]

[0138] Among them, L represents the total training loss, Indicates text loss, represents speech loss, and α and β represent weights.

[0139] Optionally, text loss Determined by the following formula:

[0140]

[0141] Among them, P θ (x t |h ＜t ) represents the autoregressive language model with training sample h ＜t Conditional on learning to transcribe text x t The target distribution, D represents the set length.

[0142] Optionally, the speech loss L a Determined by the following formula:

[0143]

[0144] Among them, P θ (s t |h ＜t ) represents the autoregressive language model with training sample h ＜t Conditional on learning speech coding samples s t The target distribution, s t The length of the corresponding audio sample is T+1, and D represents the set length.

[0145] Optionally, the speech quantizer is a kmeans model, and the device further includes a second training module, configured to perform the following steps to train the kmeans model:

[0146] Extract the intermediate representation of the 15th layer for the XLSR-53 model;

[0147] The kmeans model is trained to cluster the extracted intermediate representations to obtain discrete speech codes.

[0148] Optionally, the model training processes of the text-to-discrete speech coding module and the discrete speech coding-to-waveform module are independent of each other.

[0149] It should be noted that the device embodiment is similar to the method embodiment, so the description is relatively simple, and the relevant parts can be referred to the method embodiment.

[0150] The present application also provides an electronic device, referring to Figure 8 , Figure 8 Schematic diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the speech synthesis method disclosed in the embodiment of the present application.

[0151] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the speech synthesis method disclosed in the embodiment of the present application is implemented.

[0152] The embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the speech synthesis method disclosed in the embodiment of the present application.

[0153] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0154] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, systems, devices, storage media, and program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0158] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0159] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0160] The above is a detailed introduction to a speech synthesis method, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: The style cue speech is encoded into discrete speech codes by a pre-trained speech quantizer to obtain the cue speech code; Through a pre-trained text-to-discrete speech coding module, multiple inferences are performed based on the target text and the prompt text and prompt speech coding corresponding to the style prompt speech to obtain the target discrete speech coding corresponding to the target text; The target discrete speech code is converted into the target speech by a pre-trained discrete speech code to waveform module with the timbre prompt speech as a condition; The target discrete speech coding is determined by the following steps: According to the target text and the prompt text and prompt voice code corresponding to the style prompt voice :t′ Perform inference and obtain the decoded code s t′:t′+t-1 , where t represents the decoding step size; When the current number of inferences is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where n represents the continuous repetition threshold, w1 represents the window length, and σ represents the penalty coefficient; When the current number of inferences is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where w2 represents the window length; The probability value obtained after being reduced by σ times is divided by the temperature coefficient to obtain an updated probability distribution, and the speech code s at the current decoding time is decoded by polynomial sampling according to the updated probability distribution. t , and based on the speech coding s t The steps related to reasoning and decoding are iteratively performed until the target discrete speech code is obtained when the end code is decoded.

2. The method according to claim 1, characterized in that The text-to-discrete speech encoding module is an autoregressive language model, and the autoregressive language model is trained by the following steps: Acquire multiple audio samples from Internet data, and obtain transcription texts corresponding to each of the multiple audio samples; Encoding the plurality of audio samples into discrete speech codes by the speech quantizer to obtain a plurality of speech code samples; For each of the speech coding samples, randomly selecting a discrete speech coding not exceeding a set length from the front end of the speech coding sample as a prompt speech coding sample; For each of the speech coding samples, padding the prompt speech coding sample and the transcribed text corresponding to the speech coding sample to the set length using padding coding, processing the prompt speech coding sample and the transcribed text through an embedding layer, and adding the processed prompt speech coding sample and the transcribed text to obtain reference information; For each of the speech coding samples, the remaining discrete speech coding in the speech coding sample is processed by an embedding layer, and the reference information and the processed remaining discrete speech coding are concatenated to obtain a training sample; The autoregressive language model is trained using training samples corresponding to each of the multiple audio samples.

3. The method according to claim 2, characterized in that The reference information is determined by the following formula: c=Embedding([pad]||s 0:t )+Embedding([pad]||x) Among them, c represents reference information, Embedding represents embedding layer, [pad] represents padding encoding, || represents concatenation operation, s 0:t represents the prompt speech encoding sample, and x represents the transcribed text.

4. The method according to claim 2, characterized in that: The training sample is determined by the following formula: h=c||Embedding(s t+1: ) Among them, h represents the training sample, c represents the reference information, Embedding represents the embedding layer, || represents the concatenation operation, and s t+1: Represents the remaining discrete speech codes in the speech coding samples.

5. The method according to claim 2, characterized in that: The total training loss of the autoregressive language model during training is determined by the following formula: Among them, L represents the total training loss, Indicates text loss, represents speech loss, and α and β represent weights.

6. The method according to claim 5, characterized in that Text loss Determined by the following formula: Among them, P θ (x t |h ＜t ) represents the autoregressive language model with training sample h ＜t Conditional on learning to transcribe text x t The target distribution, D represents the set length.

7. The method according to claim 5, characterized in that Voice loss Determined by the following formula: Among them, P θ (s t |h ＜t ) represents the autoregressive language model with training sample h ＜t Conditional on learning speech coding samples s t The target distribution, s t The length of the corresponding audio sample is T+1, and D represents the set length.

8. The method according to any one of claims 1 to 7, characterized in that: The speech quantizer is a kmeans model, and the kmeans model is trained by the following steps: Extract the intermediate representation of the 15th layer for the XLSR-53 model; The kmeans model is trained to cluster the extracted intermediate representations to obtain discrete speech codes.

9. The method according to any one of claims 1 to 7, characterized in that: The model training processes of the text-to-discrete speech coding module and the discrete speech coding-to-waveform module are independent of each other.

10. A speech synthesis device, characterized in that: The device comprises: A first processing module is used to encode the style cue speech into discrete speech codes through a pre-trained speech quantizer to obtain the cue speech code; The second processing module is used to perform multiple inferences according to the target text and the prompt text and prompt voice code corresponding to the style prompt voice through a pre-trained text-to-discrete voice code module to obtain the target discrete voice code corresponding to the target text; A speech synthesis module, used for converting the target discrete speech code into a target speech with the timbre prompt speech as a condition through a pre-trained discrete speech code to waveform module; The second processing module is specifically configured to perform the following steps: According to the target text and the prompt text and prompt voice code corresponding to the style prompt voice :t′ Perform inference and obtain the decoded code s t′:t′+t-1 , where t represents the decoding step size; When the current number of inferences is less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where n represents the continuous repetition threshold, w1 represents the window length, and σ represents the penalty coefficient; When the current number of inferences is not less than n, the speech code is obtained from the probability distribution output by the text-to-discrete speech code module at the current decoding time. The probability value at the corresponding index is reduced by σ times, where w2 represents the window length; The probability value obtained after being reduced by σ times is divided by the temperature coefficient to obtain an updated probability distribution, and the speech code s at the current decoding time is decoded by polynomial sampling according to the updated probability distribution. t , and based on the speech coding s t The steps related to reasoning and decoding are iteratively performed until the target discrete speech code is obtained when the end code is decoded.

11. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the speech synthesis method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the speech synthesis method according to any one of claims 1 to 9 is implemented.

Images