Speech synthesis model training method and device, speech synthesis method and device and electronic equipment

Abstract

The invention provides a speech synthesis model training method and device, a speech synthesis method and device and electronic equipment, and the method comprises the steps: carrying out the attention calculation of the pronunciation features of a phoneme sequence and the speech features of a speech discrete mark sequence based on an initial speech synthesis model, obtaining the correlation between the pronunciation features and the speech features, and obtaining a speech discrete mark sequence based on the correlation; determining a first expected phoneme position of the current synthetic speech and a second expected phoneme position of the next synthetic speech of the current synthetic speech; determining a target loss based on the phoneme position constraint; the phoneme position constraint comprises front and back position constraint and phoneme position interval constraint between the first expected phoneme position and the second expected phoneme position; and performing parameter iteration on the initial speech synthesis model based on the target loss to obtain a speech synthesis model. According to the method, the speech synthesis training process does not depend on the duration information of the speech, the attention module of the speech synthesis model is directly constrained, and the robustness of speech synthesis is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular, to a method for training a speech synthesis model, a speech synthesis method, an apparatus, and an electronic device. Background Art

[0002] With the rapid development of Large Language Model (LLM) technology, large language models have shined in the field of speech synthesis.

[0003] When existing technical solutions such as ELLA-V and RALL-E (Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers) perform speech synthesis, they all improve robustness by introducing additional duration information to assist autoregressive prediction.

[0004] However, these methods rely on duration alignment tools to generate phoneme alignment information, and the training process is cumbersome; in addition, these methods result in a decrease in the prosody naturalness of the synthesized speech through rigid phoneme duration constraints. Summary of the Invention

[0005] The present invention provides a method for training a speech synthesis model, a speech synthesis method, an apparatus, and an electronic device, so as to solve the defects in the prior art that when performing speech synthesis based on ELLA-V and RALL-E, it relies on duration alignment tools to generate phoneme alignment information, and the training process is cumbersome; in addition, these methods result in a decrease in the prosody naturalness of the synthesized speech through rigid phoneme duration constraints.

[0006] The present invention provides a method for training a speech synthesis model, including the following steps: Obtain the phoneme sequence of the source text and the speech discrete token sequence of the target speech; Based on the initial speech synthesis model, perform attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determine the first expected phoneme position of the current synthesized speech and the second expected phoneme position of the next synthesized speech of the current synthesized speech; Based on the phoneme position constraint, determine the target loss; the phoneme position constraint includes the front-back position constraint between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraint; Perform parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

[0007] According to a training method of a speech synthesis model provided by the present invention, the phoneme position constraint further includes a start phoneme position constraint and an end phoneme position constraint; The start phoneme position constraint is used to constrain the position difference between the start phoneme position in the phoneme position sequence and the start phoneme position of the label; the phoneme position sequence is autoregressively generated in sequence based on the first expected phoneme position and the second expected phoneme position; The end phoneme position constraint is used to constrain the position difference between the end phoneme position in the phoneme position sequence and the end phoneme position of the label.

[0008] According to a training method of a speech synthesis model provided by the present invention, determining the target loss based on the phoneme position constraint includes: Determining a single-step monotonic loss based on the phoneme position constraint; Determining a prediction loss based on the prediction probability of the target speech discrete token sequence; Determining the target loss based on the single-step monotonic loss and the prediction loss.

[0009] According to a training method of a speech synthesis model provided by the present invention, the front-back position constraint is used to constrain that the first expected phoneme position is less than or equal to the second expected phoneme position; The phoneme position interval constraint is used to constrain that the position interval between the second expected phoneme position and the first expected phoneme position is less than or equal to the phoneme unit.

[0010] According to a training method of a speech synthesis model provided by the present invention, determining the first expected phoneme position of the current synthesized speech and the second expected phoneme position of the next synthesized speech of the current synthesized speech based on the relevance includes: Determining a probability matrix between the phoneme sequence and the speech discrete token sequence based on the relevance; Determining the first expected phoneme position and the second expected phoneme position based on the probability matrix.

[0011] According to a training method of a speech synthesis model provided by the present invention, determining the first expected phoneme position and the second expected phoneme position based on the probability matrix includes: Determining the first expected phoneme position and the second expected phoneme position based on the probability matrix and the phoneme position corresponding to the current synthesized speech.

[0012] The present invention also provides a speech synthesis method, including the following steps: Obtaining the text to be synthesized; Inputting the text to be synthesized into a speech synthesis model to obtain the synthesized speech output by the speech synthesis model; The speech synthesis model is obtained by executing the training method of the above speech synthesis model.

[0013] The present invention also provides a training device for a speech synthesis model, including the following units: A first acquisition unit, configured to acquire a phoneme sequence of a source text and a speech discrete token sequence of a target speech; A determining expected phoneme position unit, configured to perform attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence based on an initial speech synthesis model, obtain the correlation between the pronunciation features and the speech features, and determine a first expected phoneme position of the current synthesized speech and a second expected phoneme position of the next synthesized speech of the current synthesized speech based on the correlation; A determining target loss unit, configured to determine a target loss based on phoneme position constraints; the phoneme position constraints include the front-back position constraint between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraint; A parameter iteration unit, configured to perform parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

[0014] The present invention also provides a speech synthesis device, including the following units: A second acquisition unit, configured to acquire a text to be synthesized; An input unit, configured to input the text to be synthesized into the speech synthesis model to obtain a synthesized speech output by the speech synthesis model; The speech synthesis model is obtained by executing the training method of the above speech synthesis model.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the training method of any one of the above speech synthesis models, or implements the speech synthesis method.

[0016] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the training method of any one of the above speech synthesis models, or implements the speech synthesis method.

[0017] The present invention also provides a computer program product, including a computer program, where when the computer program is executed by a processor, it implements the training method of any one of the above speech synthesis models, or implements the speech synthesis method.

[0018] The training of the speech synthesis model, the speech synthesis method, device and electronic device provided by the present invention. On the one hand, this method does not rely on the duration information of speech for speech synthesis training, and the target loss directly constrains the attention module of the speech synthesis model. This constraint method can be used as a plug-and-play module and integrated into the existing model architecture without large-scale modification of the entire model architecture, simplifying the training process and improving the flexibility of model training. On the other hand, the target loss determined based on the phoneme position constraint can ensure the monotonicity of a single step between speech discrete tokens in the predicted target speech discrete token sequence, thereby ensuring that the synthesized speech has no missing words and no repetitions, enhancing the robustness of the synthesized speech, reducing the modeling error caused by inaccurate duration information of speech, improving the prosody naturalness of the synthesized speech, and further effectively enhancing the stability and accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic diagram of speech synthesis training based on the VALL-E model in the prior art.

[0021] Figure 2 It is a schematic diagram of speech synthesis training based on the ELLA-V model in the prior art.

[0022] Figure 3 It is a schematic diagram of speech synthesis training based on the RALL-E model in the prior art.

[0023] Figure 4 It is a difference diagram of speech synthesis training between VALL-E AR and RALL-E AR in the prior art.

[0024] Figure 5 It is a schematic flowchart of the training method of the speech synthesis model provided by the present invention.

[0025] Figure 6 It is a schematic structural diagram of the initial speech synthesis model provided by the present invention.

[0026] Figure 7 It is a schematic diagram of the attention weight matrix between the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence provided by the present invention.

[0027] Figure 8 It is a schematic flowchart of the speech synthesis method provided by the present invention.

[0028] Figure 9 It is a schematic structural diagram of a training device for a speech synthesis model provided by the present invention.

[0029] Figure 10 It is a schematic structural diagram of a speech synthesis device provided by the present invention.

[0030] Figure 11 It is a schematic structural diagram of an electronic device provided by the present invention. Specific embodiments

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0032] Terms such as "first" and "second" in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0033] In related technologies, current mainstream large speech synthesis models all adopt an autoregressive large language model modeling method to predict a semantic discrete representation sequence or an acoustic discrete representation sequence (acoustic token) from text. Figure 1 It is a schematic diagram of speech synthesis training based on the VALL-E model in the prior art. As Figure 1 shown, in the training stage of the VALL-E model, text and acoustic tokens are input into the large language model, and teacher forcing is used to predict the next acoustic token; in the inference stage, text and historical acoustic tokens are input for autoregressive prediction of the next acoustic token. This autoregressive method inherently has robustness problems, such as missing words and repetition problems during the speech generation process.

[0034] It should be noted that Figure 1Neural codec language modeling in it refers to the large language model. Text for synthesis represents the source text. The source text undergoes Phoneme conversion to obtain the phoneme sequence of the source text. 3-second encolled recording represents a 3-second speech sample of the target speaker. This recording is used to extract acoustic cue codes to help the model capture the voice characteristics of the target speaker, so as to maintain speaker consistency during speech synthesis. Audio Codec Encoder represents the audio encoder, and Audio Codec Decoder represents the audio decoder.

[0035] Existing technical solutions such as ELLA-V and RALL-E improve robustness by introducing additional duration information to assist autoregressive prediction. Figure 2 It is a schematic diagram of speech synthesis training based on the ELLA-V model in the prior art. As Figure 2 shown, ELLA-V obtains the phoneme durations of the acoustic discrete representation sequence through a phoneme duration alignment tool (Montreal Forced Aligner, MFA). During the training phase, according to the phoneme durations, the current phoneme and the end-of-phoneme marker EOP are inserted into the acoustic code, and then the phoneme and the modified acoustic token sequence are input into the large language model for training, integrating the phoneme duration information into the code prediction process.

[0036] It should be noted that the input text (Input text) is first converted into a phoneme sequence through the Phoneme Conversion module. . The input audio (Input audio) is converted into an encoded representation through the encoder (Neural Codec Encoder) of the Neural Codec Language model. The Change Sequence Order module is used to adjust the order of the phoneme sequence to meet different speech synthesis requirements or optimize the training effect of the model. The Neural Codec Language model, that is, the large language model, this module receives the adjusted phoneme sequence and the encoded representation, and generates a series of predicted acoustic tokens. Special tokens such as BOS (Beginning of Sequence) and EOP (End of Procedure) are used here to mark the start and end of the sequence. The decoder (Neural Codec Decoder) of the Neural Codec Language model converts the predicted acoustic tokens into the final synthesized speech.

[0037] Figure 3 It is a schematic diagram of speech synthesis training based on the RALL-E model in the prior art. As Figure 3 shown, RALL-E designs a chain-of-thought model. First, it predicts the duration from phonemes, then predicts the acoustic token by combining phonemes and duration, and generates the attention mask through duration guidance to ensure that the acoustic code only sees the phoneme sequence related to itself.

[0038] Figure 4 It is a difference diagram of speech synthesis training between VALL-E AR and RALL-E AR in the prior art. As Figure 4 shown, in the VALL-E model, the speech token can pay attention to all phonemes. Figure 4 The left part in shows the attention mechanism of the VALL-E model. All phonemes are allowed to be attended to by the speech token without restriction. In the RALL-E model, the speech token only pays attention to the window centered on the corresponding phoneme and prosody tokens. Figure 4 The right part in shows the attention mechanism of the RALL-E model. The speech token only pays attention to the window centered on specific phonemes and prosody tokens, and the window length is 2k + 1.

[0039] However, such methods rely on duration alignment tools to generate phoneme alignment information, and the training process is cumbersome. In addition, such methods will lead to a decrease in the prosody naturalness of the synthesized speech through hard phoneme duration constraints.

[0040] Based on the above problems, the present invention provides a training method for a speech synthesis model. Figure 5 It is a schematic flowchart of the training method for the speech synthesis model provided by the present invention. As Figure 5 shown, the method includes step 110, step 120, step 130, and step 140.

[0041] Step 110, obtain the phoneme sequence of the source text and the speech discrete token sequence of the target speech.

[0042] Specifically, obtain the phoneme sequence of the source text and the speech discrete token sequence of the target speech. Among them, the source text can be directly input by the user, or obtained by transcribing the collected audio, or obtained by collecting an image through an image acquisition device such as a scanner, mobile phone, camera, tablet, etc. and performing OCR (Optical Character Recognition) on the image. The embodiments of the present invention do not make specific limitations on this.

[0043] The phoneme sequence of the source text refers to the sequence obtained by converting the input source text (such as a sentence or a word) into phonemes. A phoneme is the basic unit of speech and represents the smallest articulatory unit in speech. For example, the word "cat" can be decomposed into the phoneme sequence / k / , / æ / , / t / .

[0044] Here, converting the source text into a phoneme sequence can be done through rule-based conversion, such as the CMU phoneme dictionary, or through a neural network-based phoneme conversion tool. The phoneme conversion tool can be g2pM or Epitran, etc. The embodiments of the present invention do not make specific limitations on this.

[0045] The speech discrete token sequence of the target speech is obtained by converting the 3-second enrolled recording (the 3-second recording of the target speaker), that is, the target speech, into a discrete token sequence.

[0046] It can be understood that the phoneme sequence mainly focuses on the abstract features of pronunciation, such as the basic attributes of speech quality, duration, intensity, and pitch, that is, the phoneme sequence is used to reflect the pronunciation features; the speech discrete token sequence of the target speech mainly focuses on acoustic details, prosodic features, and speaker features. Among them, acoustic details such as the spectral features, formants, and energy distribution of the audio, prosodic features such as intonation, speech rate, and pauses, and speaker features such as timbre, gender, and age. The embodiments of the present invention do not make specific limitations on this.

[0047] Step 120, based on the initial speech synthesis model, calculate the attention between the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence, obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determine the first expected phoneme position of the current synthesized speech and the second expected phoneme position of the next synthesized speech of the current synthesized speech.

[0048] Specifically, after obtaining the phoneme sequence and the speech discrete token sequence, based on the initial speech synthesis model, calculate the attention between the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence, obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determine the first expected phoneme position of the current synthesized speech and the second expected phoneme position of the next synthesized speech of the current synthesized speech. For example, based on the correlation, determine the probability matrix between the phoneme sequence and the speech discrete token sequence, and based on the probability matrix, determine the first expected phoneme position and the second expected phoneme position.

[0049] Here, the parameters of the initial speech synthesis model can be preset or randomly generated. The embodiments of the present invention do not make specific limitations on this.

[0050] Here, the phoneme sequence and the speech discrete token sequence can be concatenated in the time dimension and then input into the initial speech synthesis model. The initial speech synthesis model performs attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features. Figure 6 is a schematic structural diagram of the initial speech synthesis model provided by the present invention, as Figure 6 shown, the initial speech synthesis model may include multiple layers of Transformer models, and the multiple layers of Transformer models perform attention calculation between the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence. The phoneme sequence can be represented by and the speech discrete token sequence can be represented by , and the output of the initial speech synthesis model is .

[0051] It should be noted that the initial speech synthesis model may be a large language model, and the large language model may be a pre-trained language model such as the IFlytek Spark model, the XLNet model (extreme Multi-label Learning Network), the ROBERTa model (Robustly Optimized BERT approach), etc. The embodiments of the present invention do not make specific limitations on this.

[0052] Step 130, determining a target loss based on phoneme position constraints; the phoneme position constraints include the front-back position constraint between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraint.

[0053] Specifically, considering that in speech processing, based on speech prior knowledge, it can be determined that one phoneme pronunciation corresponds to multiple speech codes, and the relationship between the phoneme pronunciation and the speech code can be constrained and optimized through the "single-step monotonic" characteristic. Specifically, single-step monotonic means that in the process of speech synthesis or recognition, the expected phoneme position of each speech code needs to satisfy a monotonically increasing order, that is, the state of each code can only be a continuation of the current phoneme or jump to the next phoneme, but cannot skip any phoneme.

[0054] Correspondingly, a target loss can be determined based on the phoneme position constraints, where the phoneme position constraints include the front-back position constraint between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraint.

[0055] Here, the front-back position constraint is used to constrain the first expected phoneme position to be less than or equal to the second expected phoneme position , the phoneme position interval constraint is used to constrain the second expected phoneme position and the first expected phoneme position such that the position interval is less than or equal to the phoneme unit.

[0056] It should be noted that the first expected phoneme position refers to the expected phoneme position of the current code, and the second expected phoneme position refers to the expected phoneme position of the next code. Correspondingly, the front-back position constraint between the first expected phoneme position and the second expected phoneme position means that the expected phoneme position of the current code is less than or equal to the expected phoneme position of the next code , and the phoneme position interval constraint between the first expected phoneme position and the second expected phoneme position means that the expected phoneme position of the next code relative to the expected phoneme position of the current code will not exceed one phoneme.

[0057] Correspondingly, denotes the phoneme position interval between the expected phoneme position of the current code and the expected phoneme position of the next code , and represents that the expected phoneme position of the next code relative to the expected phoneme position of the current code will not exceed one phoneme.

[0058] Here, the front-back position constraint between the first expected phoneme position and the second expected phoneme position can be represented by , and the phoneme position interval constraint between the first expected phoneme position and the second expected phoneme position can be represented by .

[0059] It can be understood that based on the phoneme position constraint, the target loss is determined. The phoneme position constraint includes the front-back position constraint between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraint. On the one hand, since the phonemes in speech are arranged in a certain order, the single-step monotonic constraint can ensure that there will be no confusion or jump in the phoneme order during the processing, thus ensuring the orderliness of the phonemes and further improving the naturalness of the synthesized speech rhythm; on the other hand, by restricting that the state of each encoding can only be a continuation of the current phoneme or a jump to the next phoneme, the model can more accurately align the phonemes and the speech encoding, thereby improving the accuracy and interpretability of speech synthesis.

[0060] It should be understood that the greater the positional constraint before and after between the first expected phoneme position and the second expected phoneme position, and the greater the phoneme position interval constraint, the greater the target loss; the smaller the positional constraint before and after between the first expected phoneme position and the second expected phoneme position, and the smaller the phoneme position interval constraint, the smaller the target loss.

[0061] Step 140, perform parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

[0062] Specifically, after obtaining the target loss, parameter iteration can be performed on the initial speech synthesis model based on the target loss, and the initial speech synthesis model after completing the parameter iteration is used as the speech synthesis model.

[0063] The method provided by the embodiments of the present invention obtains the phoneme sequence of the source text and the speech discrete token sequence of the target speech, then based on the initial speech synthesis model, calculates the attention between the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determines the first expected phoneme position of the current synthesized speech and the second expected phoneme position of the next synthesized speech of the current synthesized speech. Then, based on the phoneme position constraint, the target loss is determined; the phoneme position constraint includes the positional constraint before and after between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraint. Finally, parameter iteration is performed on the initial speech synthesis model based on the target loss to obtain the speech synthesis model. On the one hand, this method does not rely on the duration information of the speech for speech synthesis training, and the target loss directly constrains the attention module of the speech synthesis model. This constraint method can be used as a plug-and-play module and integrated into the existing model architecture without large-scale modification of the entire model architecture, simplifying the training process and improving the flexibility of model training. On the other hand, the target loss determined based on the phoneme position constraint can ensure the monotonicity of the speech discrete tokens in the predicted target speech discrete token sequence step by step, thereby ensuring that the synthesized speech does not miss words or repeat, improving the robustness of the synthesized speech, reducing the modeling error caused by inaccurate duration information of the speech, enhancing the prosody naturalness of the synthesized speech, and further effectively improving the stability and accuracy of speech synthesis.

[0064] Based on the above embodiments, the phoneme position constraint further includes a head phoneme position constraint and a tail phoneme position constraint; The head phoneme position constraint is used to constrain the positional difference between the starting phoneme position in the phoneme position sequence and the head phoneme position of the label; the phoneme position sequence is autoregressively generated in sequence based on the first expected phoneme position and the second expected phoneme position; The end phoneme position constraint is used to constrain the position difference between the ending phoneme position in the phoneme position sequence and the end phoneme position of the label.

[0065] Specifically, the phoneme position constraint further includes a start phoneme position constraint and an end phoneme position constraint. Among them, the start phoneme position constraint is used to constrain the position difference between the starting phoneme position in the phoneme position sequence and the start phoneme position of the label. The phoneme position sequence is autoregressively generated in sequence based on the first expected phoneme position and the second expected phoneme position.

[0066] The end phoneme position constraint is used to constrain the position difference between the ending phoneme position in the phoneme position sequence and the end phoneme position of the label.

[0067] Here, the start phoneme position of the label and the end phoneme position of the label are as follows: Among them, 1 represents the start phoneme position of the label, represents the end phoneme position of the label.

[0068] It should be noted that the start phoneme position constraint and the end phoneme position constraint require that the phoneme position sequence starts from the first phoneme to the last phoneme.

[0069] To sum up, the phoneme position constraint includes the front - back position constraint, the phoneme position interval constraint, the start phoneme position constraint, and the end phoneme position constraint between the first expected phoneme position and the second expected phoneme position.

[0070] Correspondingly, based on the phoneme position constraint, the formula for determining the target loss is as follows: Among them, represents the target loss, represents the front - back position constraint between the first expected phoneme position and the second expected phoneme position, represents the phoneme position interval constraint between the first expected phoneme position and the second expected phoneme position, represents the start phoneme position constraint, represents the end phoneme position constraint.

[0071] It can be understood that represents the single - step monotonic loss. By adding a single - step monotonic constraint to the weight matrix of the attention module in the speech synthesis model, it is ensured that the speech discrete markers in the predicted target speech discrete marker sequence are single - step monotonic, thereby ensuring that the synthesized speech has no missing words and no repetitions, and improving the robustness of the synthesized speech.

[0072] In the method provided by the embodiment of the present invention, the phoneme position constraint further includes a start phoneme position constraint and an end phoneme position constraint. The start phoneme position constraint is used to constrain the position difference between the start phoneme position in the phoneme position sequence and the start phoneme position of the label. The phoneme position sequence is autoregressively generated in sequence based on the first expected phoneme position and the second expected phoneme position; the end phoneme position constraint is used to constrain the position difference between the end phoneme position in the phoneme position sequence and the end phoneme position of the label. The start phoneme position constraint and the end phoneme position constraint can clarify the specific boundaries of each phoneme in the phoneme position sequence, thereby reducing the ambiguity caused by fuzzy phoneme boundaries and improving the model's ability to model phoneme duration.

[0073] Based on the above embodiment, step 130 includes: Step 131, determining a single-step monotonic loss based on the phoneme position constraint; Step 132, determining a prediction loss based on the prediction probability of the target speech discrete token sequence; Step 133, determining the target loss based on the single-step monotonic loss and the prediction loss.

[0074] Specifically, a single-step monotonic loss can be determined based on the phoneme position constraint, that is, a single-step monotonic loss can be determined based on the front-back position constraint, the phoneme position interval constraint, the start phoneme position constraint, and the end phoneme position constraint between the first expected phoneme position and the second expected phoneme position. The formula for the single-step monotonic loss is as follows: Among them, represents the single-step monotonic loss, represents the front-back position constraint between the first expected phoneme position and the second expected phoneme position, represents the phoneme position interval constraint between the first expected phoneme position and the second expected phoneme position, represents the start phoneme position constraint, represents the end phoneme position constraint.

[0075] Here, the prediction probability of the target speech discrete token sequence is as follows: Among them, represents the prediction probability of the target speech discrete token sequence, represents the phoneme sequence of the source text, represents the speech discrete token sequence of the target speech.

[0076] It should be noted that the target speech discrete token sequence is the speech discrete token sequence finally output by the speech synthesis model. .

[0077] Correspondingly, based on the prediction probability of the target speech discrete token sequence, the formula for determining the prediction loss is as follows: Wherein, represents the prediction loss, represents the prediction probability of the target speech discrete token sequence, that is, the likelihood probability of the target speech discrete token sequence.

[0078] Finally, after obtaining the single-step monotonic loss and the prediction loss, the target loss can be determined based on the single-step monotonic loss and the prediction loss. The formula for the target loss is as follows: Wherein, represents the target loss, represents the prediction loss, represents the single-step monotonic loss, represents the weight coefficient.

[0079] The method provided by the embodiments of the present invention determines the single-step monotonic loss based on the phoneme position constraint, then determines the prediction loss based on the prediction probability of the target speech discrete token sequence, and finally determines the target loss based on the single-step monotonic loss and the prediction loss. On the one hand, the single-step monotonic loss can effectively avoid the repeated generation or omission of phonemes by constraining the positions of phonemes, thereby reducing errors in speech synthesis and optimizing the generation order of phonemes; on the other hand, determining the prediction loss based on the prediction probability of the target speech discrete token sequence can constrain the output of the model to make it closer to the true synthetic speech distribution, thereby further improving the accuracy of speech synthesis.

[0080] Based on the above embodiments, the step of determining the first expected phoneme position of the current synthetic speech and the second expected phoneme position of the next synthetic speech of the current synthetic speech based on the relevance in step 120 includes: Step 121, determining the probability matrix between the phoneme sequence and the speech discrete token sequence based on the relevance; Step 122, determining the first expected phoneme position and the second expected phoneme position based on the probability matrix.

[0081] Specifically, without loss of generality, in the embodiments of the present invention, any layer of Transformer is taken as an example to detail the method of the present invention. The attention weight matrix (correlation) between the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence in the attention weights of this layer of Transformer is taken out. After passing through the softmax activation function, a probability matrix can be obtained in which each code in the phoneme position sequence corresponds to each phoneme. , where , represents the probability that the j-th code in the phoneme position sequence corresponds to the i-th phoneme. Furthermore, the expected phoneme position of the j-th code in the phoneme position sequence can be calculated, that is, the first expected phoneme position and the second expected phoneme position can be determined.

[0082] Figure 7 FIG. is a schematic diagram of the attention weight matrix between the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence provided by the present invention. As Figure 7 shown, the attention weight matrix between the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence corresponds to the light yellow part of the matrix. It can be understood that in the autoregressive training of the speech synthesis model, the speech synthesis model needs to gradually generate the target speech discrete token sequence while considering the previously generated speech discrete tokens. The matrix in the light yellow part represents how the model uses the alignment information between the pronunciation features of the previous phoneme sequence and the speech features of the speech discrete token sequence to guide the generation of the current speech discrete token during the generation process, thereby improving the stability of training and the accuracy of generation.

[0083] In the autoregressive training of the speech synthesis model, the speech synthesis model needs to gradually generate each element of the sequence. To ensure the rationality of the generation process, when generating the t-th element, the speech synthesis model can only utilize the information before the t-th element (including the t-th element), and cannot utilize the information after the t-th element. This mechanism is called causal masking (Causal Masking). Correspondingly, Figure 6 the blue part in is the causal masking area.

[0084] Based on the above embodiments, step 122 includes: Based on the probability matrix and the phoneme position corresponding to the current synthesized speech, determine the first expected phoneme position and the second expected phoneme position.

[0085] Specifically, the first expected phoneme position and the second expected phoneme position can be determined based on the probability matrix and the phoneme position corresponding to the current synthesized speech. The formula is as follows: where represents a probability matrix, represents the phoneme position corresponding to the current synthesized speech, represents the first expected phoneme position, and the formula for the second expected phoneme position is similar and will not be elaborated here.

[0086] Considering that after training the speech synthesis model, the speech synthesis model can be applied for speech synthesis.

[0087] Based on any of the above embodiments, the present invention provides a method for training a speech synthesis model, and the method includes: First, obtain the phoneme sequence of the source text and the speech discrete token sequence of the target speech.

[0088] Then, based on the initial speech synthesis model, perform attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determine the probability matrix between the phoneme sequence and the speech discrete token sequence, and then based on the probability matrix and the phoneme position corresponding to the current synthesized speech, determine the first expected phoneme position and the second expected phoneme position.

[0089] Further, based on the phoneme position constraint, determine the single-step monotonic loss, where the phoneme position constraint includes the front-back position constraint between the first expected phoneme position and the second expected phoneme position, as well as the phoneme position interval constraint, the start phoneme position constraint, and the end phoneme position constraint. Here, the front-back position constraint is used to constrain that the first expected phoneme position is less than or equal to the second expected phoneme position; the phoneme position interval constraint is used to constrain that the position interval between the second expected phoneme position and the first expected phoneme position is less than or equal to the phoneme unit. The start phoneme position constraint is used to constrain the position difference between the start phoneme position in the phoneme position sequence and the start phoneme position of the label; the phoneme position sequence is autoregressively generated in sequence based on the first expected phoneme position and the second expected phoneme position. The end phoneme position constraint is used to constrain the position difference between the end phoneme position in the phoneme position sequence and the end phoneme position of the label.

[0090] Based on the prediction probability of the target speech discrete token sequence, determine the prediction loss, and then based on the single-step monotonic loss and the prediction loss, determine the target loss.

[0091] Finally, perform parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

[0092] Based on any of the above embodiments, the present invention provides a speech synthesis method, Figure 8 is a schematic flowchart of the speech synthesis method provided by the present invention, as Figure 8 shown, the method includes: Step 810, obtain the text to be synthesized; Step 820: Input the text to be synthesized into the speech synthesis model to obtain the synthesized speech output by the speech synthesis model. The speech synthesis model is obtained by executing the training method of the above speech synthesis model.

[0093] Specifically, first, obtain the text to be synthesized. Here, the text to be synthesized refers to the text that needs to be synthesized into speech subsequently. The text to be synthesized can be directly input by the user, or obtained by transcribing the collected audio into speech, or obtained by collecting images through image acquisition devices such as scanners, mobile phones, cameras, tablets, etc. and performing OCR on the images. The embodiments of the present invention do not make specific limitations on this.

[0094] After obtaining the text to be synthesized, the text to be synthesized can be input into the speech synthesis model to obtain the synthesized speech output by the speech synthesis model. Among them, the speech synthesis model is trained based on the phoneme sequence of the source text and the speech discrete token sequence of the target speech.

[0095] It can be understood that the phoneme sequence mainly focuses on the abstract features of pronunciation. For example, the basic attributes such as the quality, length, intensity, and pitch of the speech, that is, the phoneme sequence is used to reflect the pronunciation features; the speech discrete token sequence of the target speech mainly focuses on the acoustic details, prosodic features, and speaker features. Among them, the acoustic details such as the spectral features, formants, and energy distribution of the audio, the prosodic features such as intonation, speech rate, and pause, and the speaker features such as timbre, gender, and age. The embodiments of the present invention do not make specific limitations on this.

[0096] Specifically, based on the initial speech synthesis model, attention calculation can be performed on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determine the first expected phoneme position of the current synthesized speech and the second expected phoneme position of the next synthesized speech of the current synthesized speech. For example, based on the correlation, a probability matrix between the phoneme sequence and the speech discrete token sequence can be determined, and based on the probability matrix, the first expected phoneme position and the second expected phoneme position can be determined.

[0097] Here, the phoneme sequence and the speech discrete token sequence can be concatenated in the time dimension and then input into the initial speech synthesis model, and the initial speech synthesis model performs attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features.

[0098] It should be noted that the initial speech synthesis model can be a large language model, and the large language model can be a pre-trained language model such as iFlytek Spark Model, XLNet model, ROBERTa model, etc. The embodiments of the present invention do not make specific limitations in this regard.

[0099] Then, based on the phoneme position constraints, a single-step monotonic loss is determined. Among them, the phoneme position constraints include the front-back position constraints between the first expected phoneme position and the second expected phoneme position, as well as the phoneme position interval constraints, the start phoneme position constraints, and the end phoneme position constraints. Then, based on the prediction probability of the target speech discrete token sequence, a prediction loss is determined. Finally, based on the single-step monotonic loss and the prediction loss, a target loss is determined.

[0100] After obtaining the target loss, the initial speech synthesis model can be iteratively parameterized based on the target loss, and the initial speech synthesis model after completing the parameter iteration is used as the speech synthesis model.

[0101] The method provided by the embodiments of the present invention obtains the text to be synthesized, inputs the text to be synthesized into the speech synthesis model, and obtains the synthesized speech output by the speech synthesis model. The speech synthesis model is obtained by executing the above-mentioned training method of the speech synthesis model. On the one hand, by constraining the positions of phonemes, the single-step monotonic loss can effectively avoid the repeated generation or omission of phonemes, thereby reducing errors in speech synthesis, optimizing the generation order of phonemes, and improving the naturalness of the prosody of the synthesized speech. On the other hand, determining the prediction loss based on the prediction probability of the target speech discrete token sequence can constrain the output of the model to make it closer to the true synthesized speech distribution, thereby further improving the accuracy of speech synthesis.

[0102] Next, the training device of the speech synthesis model provided by the present invention will be described. The training device of the speech synthesis model described below can be mutually corresponded and referred to the training method of the speech synthesis model described above.

[0103] Based on any of the above embodiments, the present invention provides a training device for a speech synthesis model. Figure 9 It is a schematic structural diagram of the training device for the speech synthesis model provided by the present invention, as Figure 9 shown. The device includes: The first acquisition unit 910 is used to acquire the phoneme sequence of the source text and the speech discrete token sequence of the target speech; The unit 920 for determining the expected phoneme position is used to perform attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence based on the initial speech synthesis model, obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determine the first expected phoneme position of the current synthesized speech and the second expected phoneme position of the next synthesized speech of the current synthesized speech; Determine a target loss unit 930 for determining a target loss based on phoneme position constraints; the phoneme position constraints include a front-back position constraint between the first expected phoneme position and the second expected phoneme position, and a phoneme position interval constraint; A parameter iteration unit 940 is configured to perform parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

[0104] For the device provided by the embodiment of the present invention, on the one hand, this method does not rely on the duration information of speech for speech synthesis training. The target loss directly constrains the attention module of the speech synthesis model. This constraint method can be used as a plug-and-play module and integrated into the existing model architecture without large-scale modification of the entire model architecture, simplifying the training process and improving the flexibility of model training. On the other hand, the target loss determined based on the phoneme position constraints can ensure the monotonicity of a single step between speech discrete markers in the predicted target speech discrete marker sequence, thereby ensuring that the synthesized speech has no missing words and no repetitions, improving the robustness of the synthesized speech, reducing the modeling error caused by inaccurate duration information of speech, enhancing the prosodic naturalness of the synthesized speech, and further effectively improving the stability and accuracy of speech synthesis.

[0105] Based on any of the above embodiments, the phoneme position constraints further include a start phoneme position constraint and an end phoneme position constraint; The start phoneme position constraint is used to constrain the position difference between the start phoneme position in the phoneme position sequence and the start phoneme position of the label; the phoneme position sequence is autoregressively generated in sequence based on the first expected phoneme position and the second expected phoneme position; The end phoneme position constraint is used to constrain the position difference between the end phoneme position in the phoneme position sequence and the end phoneme position of the label.

[0106] Based on any of the above embodiments, the target loss unit 930 is specifically configured to: Determine a single-step monotonic loss based on the phoneme position constraints; Determine a prediction loss based on the prediction probability of the target speech discrete marker sequence; Determine the target loss based on the single-step monotonic loss and the prediction loss.

[0107] Based on any of the above embodiments, the front-back position constraint is used to constrain that the first expected phoneme position is less than or equal to the second expected phoneme position; The phoneme position interval constraint is used to constrain that the position interval between the second expected phoneme position and the first expected phoneme position is less than or equal to a phoneme unit.

[0108] Based on any of the above embodiments, the determining expected phoneme position unit 920 specifically includes: A determining probability matrix unit, configured to determine a probability matrix between the phoneme sequence and the speech discrete token sequence based on the relevance; A determining expected phoneme position subunit, configured to determine the first expected phoneme position and the second expected phoneme position based on the probability matrix.

[0109] Based on any of the above embodiments, the determining expected phoneme position subunit specifically is configured to: Determine the first expected phoneme position and the second expected phoneme position based on the probability matrix and the phoneme position corresponding to the current synthesized speech.

[0110] The speech synthesis device provided by the present invention will be described below. The speech synthesis device described below can be correspondingly referred to the speech synthesis method described above.

[0111] Based on any of the above embodiments, the present invention provides a speech synthesis device, Figure 10 which is a schematic structural diagram of the speech synthesis device provided by the present invention, as Figure 10 shown. The device includes: A second obtaining unit 1010, configured to obtain a text to be synthesized; An input unit 1020, configured to input the text to be synthesized into a speech synthesis model to obtain a synthesized speech output by the speech synthesis model; The speech synthesis model is obtained by executing based on the training method of the above speech synthesis model.

[0112] The device provided by the embodiment of the present invention obtains a text to be synthesized, inputs the text to be synthesized into a speech synthesis model to obtain a synthesized speech output by the speech synthesis model, and the speech synthesis model is obtained by executing based on the training method of the above speech synthesis model. On the one hand, the single-step monotonic loss can effectively avoid the repeated generation or omission of phonemes by constraining the positions of phonemes, thereby reducing errors in speech synthesis and optimizing the generation order of phonemes; on the other hand, determining the prediction loss based on the prediction probability of the target speech discrete token sequence can constrain the output of the model to make it closer to the true synthesized speech distribution, thereby further improving the accuracy of speech synthesis.

[0113] Figure 11 which is a schematic structural diagram of the electronic device provided by the present invention, as Figure 11As shown in the figure, the electronic device may include: a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140. Among them, the processor 1110, the communications interface 1120, and the memory 1130 complete communication with each other through the communication bus 1140. The processor 1110 may call the logical instructions in the memory 1130 to execute the training method of the speech synthesis model. The method includes: obtaining a phoneme sequence of the source text and a speech discrete token sequence of the target speech; based on the initial speech synthesis model, performing attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determining the first expected phoneme position of the current synthesized speech and the second expected phoneme position of the next synthesized speech of the current synthesized speech; determining a target loss based on phoneme position constraints; the phoneme position constraints include the front-back position constraints between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraints; performing parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

[0114] The processor 1110 may also call the logical instructions in the memory 1130 to execute the speech synthesis method. The method includes: obtaining the text to be synthesized; inputting the text to be synthesized into the speech synthesis model to obtain the synthesized speech output by the speech synthesis model; the speech synthesis model is obtained by executing the training method of the above speech synthesis model.

[0115] In addition, when the logical instructions in the above-mentioned memory 1130 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0116] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the speech synthesis model provided by each of the above methods. The method includes: obtaining a phoneme sequence of a source text and a speech discrete token sequence of a target speech; based on an initial speech synthesis model, performing attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determining a first expected phoneme position of the current synthesized speech and a second expected phoneme position of the next synthesized speech of the current synthesized speech; determining a target loss based on phoneme position constraints; the phoneme position constraints include the front-back position constraint between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraint; performing parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

[0117] When the computer program is executed by a processor, the computer can execute the speech synthesis method provided by each of the above methods. The method includes: obtaining a text to be synthesized; inputting the text to be synthesized into a speech synthesis model to obtain a synthesized speech output by the speech synthesis model; the speech synthesis model is obtained by executing the training method of the speech synthesis model described above.

[0118] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the training method of the speech synthesis model provided by each of the above methods. The method includes: obtaining a phoneme sequence of a source text and a speech discrete token sequence of a target speech; based on an initial speech synthesis model, performing attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain the correlation between the pronunciation features and the speech features, and based on the correlation, determining a first expected phoneme position of the current synthesized speech and a second expected phoneme position of the next synthesized speech of the current synthesized speech; determining a target loss based on phoneme position constraints; the phoneme position constraints include the front-back position constraint between the first expected phoneme position and the second expected phoneme position, and the phoneme position interval constraint; performing parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

[0119] When the computer program is executed by a processor, it implements a speech synthesis method provided by the above-mentioned various methods. The method includes: obtaining text to be synthesized; inputting the text to be synthesized into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; the speech synthesis model is obtained by executing based on the training method of the above-mentioned speech synthesis model.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0121] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a speech synthesis model, characterized in that Including: Obtaining a phoneme sequence of a source text and a speech discrete token sequence of a target speech; Based on an initial speech synthesis model, performing attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence to obtain a correlation degree between the pronunciation features and the speech features, and based on the correlation degree, determining a first expected phoneme position of a current synthesized speech and a second expected phoneme position of a next synthesized speech of the current synthesized speech; Determining a target loss based on phoneme position constraints; the phoneme position constraints include a front-back position constraint between the first expected phoneme position and the second expected phoneme position, and a phoneme position interval constraint; Performing parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

2. The training method of the speech synthesis model according to claim 1, characterized in that The phoneme position constraints further include a start phoneme position constraint and an end phoneme position constraint; The start phoneme position constraint is used to constrain the position difference between the start phoneme position in a phoneme position sequence and the start phoneme position of a label; the phoneme position sequence is autoregressively generated in sequence based on the first expected phoneme position and the second expected phoneme position; The end phoneme position constraint is used to constrain the position difference between the end phoneme position in the phoneme position sequence and the end phoneme position of a label.

3. The training method of the speech synthesis model according to claim 1, wherein The determining a target loss based on phoneme position constraints includes: Determining a single-step monotonic loss based on the phoneme position constraints; Determining a prediction loss based on the prediction probability of a target speech discrete token sequence; Determining the target loss based on the single-step monotonic loss and the prediction loss.

4. The training method of the speech synthesis model according to any one of claims 1 to 3, characterized in that The front-back position constraint is used to constrain that the first expected phoneme position is less than or equal to the second expected phoneme position; The phoneme position interval constraint is used to constrain that the position interval between the second expected phoneme position and the first expected phoneme position is less than or equal to a phoneme unit.

5. The training method of the speech synthesis model according to any one of claims 1 to 3, characterized in that The determining the first expected phoneme position of a current synthesized speech and the second expected phoneme position of a next synthesized speech of the current synthesized speech based on the correlation degree includes: Determining a probability matrix between the phoneme sequence and the speech discrete token sequence based on the correlation degree; Determining the first expected phoneme position and the second expected phoneme position based on the probability matrix.

6. The training method of the speech synthesis model according to claim 5, characterized in that, The determining the first expected phoneme position and the second expected phoneme position based on the probability matrix includes: Determining the first expected phoneme position and the second expected phoneme position based on the probability matrix and the phoneme position corresponding to the current synthesized speech.

7. A voice synthesis method, characterized in that, Including: Obtaining a text to be synthesized; Inputting the text to be synthesized into a speech synthesis model to obtain a synthesized speech output by the speech synthesis model; The speech synthesis model is obtained by executing the training method of the speech synthesis model according to any one of claims 1 to 6.

8. A training device for a speech synthesis model, characterized in that Including: A first obtaining unit, configured to obtain a phoneme sequence of a source text and a speech discrete token sequence of a target speech; A desired phoneme position determination unit, configured to perform attention calculation on the pronunciation features of the phoneme sequence and the speech features of the speech discrete token sequence based on an initial speech synthesis model, obtain the correlation between the pronunciation features and the speech features, and determine a first desired phoneme position of the current synthesized speech and a second desired phoneme position of the next synthesized speech of the current synthesized speech based on the correlation; A target loss determination unit, configured to determine a target loss based on phoneme position constraints; the phoneme position constraints include the front-back position constraint between the first desired phoneme position and the second desired phoneme position, and the phoneme position interval constraint; A parameter iteration unit, configured to perform parameter iteration on the initial speech synthesis model based on the target loss to obtain the speech synthesis model.

9. A voice synthesis device, characterized in that, Comprising: A second acquisition unit, configured to acquire text to be synthesized; An input unit, configured to input the text to be synthesized into the speech synthesis model to obtain a synthesized speech output by the speech synthesis model; The speech synthesis model is obtained by executing the training method of the speech synthesis model according to any one of claims 1 to 6.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the training method of the speech synthesis model according to any one of claims 1 to 6, or implements the speech synthesis method according to claim 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the speech synthesis model according to any one of claims 1 to 6, or implements the speech synthesis method according to claim 7.

Images