Speech synthesis method, device, electronic device, storage medium and program product

Abstract

The present application discloses a speech synthesis method, apparatus, electronic device, storage medium, and program product for improving the efficiency of model-based speech synthesis and meeting the needs of real-time speech synthesis. The speech synthesis method includes: performing speech conversion on a first text based on a first sequence using a first model to obtain a first speech unit sequence, wherein the first sequence includes text units of the first text; performing speech conversion on the first text based on the first speech unit sequence and the first sequence using a second model to obtain a second speech unit sequence; the parameters of the first model are fewer than the parameters of the second model; generating a third speech unit sequence based on the matching results between the first speech unit sequence and the second speech unit sequence; and generating synthesized speech of the first text based on the third speech unit sequence.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method, device, electronic device, storage medium and program product. Background Art

[0002] Large language model (LLM) technology is booming, and phenomenal products have emerged in the field of natural language processing (NLP), enabling realistic online text conversations that are indistinguishable from real people. The main bottleneck of LLM technology lies in its enormous computing power requirements. Model inference requires a large number of floating-point calculations, and the longer the sequence to be generated, the greater the computing power required.

[0003] Text-to-speech (TTS) technology based on large language models, particularly those based on large autoregressive (AR) language models, is also limited by computing power, making it difficult to apply to real-time speech generation services. Compared to large language models in the NLP field, large TTS language models take longer to infer, resulting in lower speech synthesis efficiency.

[0004] Therefore, how to improve the efficiency of speech synthesis based on large language models and meet the needs of real-time speech synthesis has become a hot research direction. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a speech synthesis method, device, electronic device, storage medium and program product for improving the efficiency of model-based speech synthesis and meeting the needs of real-time speech synthesis.

[0006] In order to achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a speech synthesis method, comprising:

[0008] Performing speech conversion on a first text based on a first sequence using a first model to obtain a first speech unit sequence, where the first sequence includes text units of the first text;

[0009] performing speech conversion on the first text based on the first speech unit sequence and the first sequence using a second model to obtain a second speech unit sequence; the parameters of the first language model are less than the parameters of the second model;

[0010] generating a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence;

[0011] Based on the third speech unit sequence, synthesized speech of the first text is generated.

[0012] In a second aspect, an embodiment of the present application provides a speech synthesis device, comprising:

[0013] a first generating module, configured to perform speech conversion on a first text based on a first sequence using a first model to obtain a first speech unit sequence, where the first sequence includes text units of the first text;

[0014] a second generating module, configured to perform speech conversion on the first text based on the first speech unit sequence and the first sequence using a second model to obtain a second speech unit sequence; wherein the parameters of the first model are less than the parameters of the second model;

[0015] a third generating module, configured to generate a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence;

[0016] A speech synthesis module is used to generate synthesized speech of the first text based on the third speech unit sequence.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0018] processor;

[0019] a memory for storing instructions executable by the processor;

[0020] The processor is configured to execute the instructions to implement the speech synthesis method provided in the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the speech synthesis method provided in the second aspect.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute some or all of the steps in the speech synthesis method of the first aspect.

[0023] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0024] This method uses speculative sampling as a technical approach and employs a speech synthesis solution that utilizes both primary and secondary models. Specifically, a lightweight model with fewer parameters is first used to perform speech conversion on the first text based on a first sequence of text units containing the first text, resulting in a first speech unit sequence. A heavyweight model with more parameters is then used to perform speech conversion on the first text using the first sequence and the first speech unit sequence, resulting in a second speech unit sequence. The second speech unit sequence is then used to verify the first speech unit sequence, identifying reliable speech units for adoption. Furthermore, the above steps are repeated, starting with the speech units that were not adopted, until all speech units corresponding to the first text are obtained, and the synthesized speech of the text is obtained based on these speech units.

[0025] Since the parameters of the lightweight model are much smaller than those of the heavyweight model, the prediction time of the lightweight model is much shorter than that of the heavyweight model; secondly, since the large model uses the obtained speech units to predict the next speech unit and uses matrix calculation, based on the speech unit sequence output by the existing lightweight model, the heavyweight model can complete the verification of the speech unit sequence output by the lightweight model after one prediction, which is significantly reduced compared to the traditional multi-step prediction scheme. Moreover, although the reliability of the lightweight model is not as good as that of the heavyweight model, the quality of the verified speech units can be ensured after verification by the heavyweight model. It can be seen that the speech synthesis method proposed in the embodiment of the present application significantly improves the efficiency of speech synthesis while ensuring the accuracy of speech synthesis, and can meet the needs of real-time speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 A flowchart of a speech synthesis method provided in one embodiment of the present application;

[0028] Figure 2 A schematic flow chart of a first model training method provided in one embodiment of the present application;

[0029] Figure 3 A schematic diagram of training data of a second model provided in one embodiment of the present application;

[0030] Figure 4 A schematic structural diagram of a second model provided for one embodiment of the present application;

[0031] Figure 5 A schematic structural diagram of a first model provided for one embodiment of the present application;

[0032] Figure 6 A schematic structural diagram of a first model provided for another embodiment of the present application;

[0033] Figure 7 A schematic diagram of a method for acquiring a second sample sequence provided in one embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of a speech synthesis device provided in one embodiment of the present application;

[0035] Figure 9 A schematic structural diagram of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0036] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] The terms "first," "second," and the like in this specification and claims are used to distinguish similar objects and are not intended to describe a particular order or precedence. It should be understood that such terms are interchangeable where appropriate so that the embodiments of the present application can be implemented in sequences other than those illustrated or described herein. In addition, the term "and / or" in this specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the connected objects are in an "or" relationship.

[0038] Explanation of relevant terms:

[0039] Large language models: These are large-scale language models trained on high-computing platforms using large amounts of training data. Self-supervised or semi-supervised training is typically used for large language models.

[0040] Autoregressive Large Language Model (ALM): A method for constructing LLMs, currently the mainstream LLM technology. The ALM utilizes a decoder-only architecture and left-to-right unidirectional prediction training consistent with natural language characteristics. During model inference, each inference step predicts only one unit (token). This predicted unit is appended to the end of the input sequence and serves as the input for the next unit prediction.

[0041] Based on a large TTS language model, this system uses large-scale parameters trained on a high-computing platform based on a large amount of speech data. Its characteristic is that it uses speech unitization technology to convert continuous speech signals into discrete speech units (tokens). Then, speech units and text units are treated as equal units, thereby constructing an LLM large language model that generates speech units from text units within a single language model.

[0042] Speculative sampling: A technical approach involves training a smaller model with fewer parameters in addition to the LLM. This smaller model may have lower accuracy than the LLM, but is faster. Therefore, using a smaller model for multiple sampling steps can be faster than using the LLM for a single sampling step.

[0043] Transformer: It is the foundation of the current large language model. Its multi-layer stacked model structure based on multi-head attention constitutes the structural main body of modern LLM.

[0044] As mentioned earlier, large TTS language models are also limited by computing power, making them difficult to apply to real-time speech generation services. Compared to large language models in the NLP field, large TTS language models take longer to infer, resulting in lower speech synthesis efficiency. This is because, generally speaking, the bitrate of quantized speech signals is higher than the bitrate of text. For example, taking the text "I like this color," assuming one character is one code, the bitrate for this text is 7 text tokens per second. However, for the speech of this text, assuming the speech lasts one second, at a 16k sampling rate, this translates to 16,000 samples. Typical speech codec (codec) technology can compress this speech to 30-50 tokens, with a maximum of around 20 tokens. This results in a bitrate of 20 speech tokens per second, far higher than the bitrate for text. In TTS scenarios, speech tokens are precisely the target of large language model generation. Therefore, TTS technology based on large language models urgently needs to improve speech synthesis speed to meet the requirements of real-time speech synthesis.

[0045] The applicant conducted research on traditional TTS technology based on large language models and found that the main reason for the low efficiency of speech synthesis is that the large language model needs to go through multiple iterative predictions during the process of language synthesis based on the input sequence of text tokens containing text, that is, in the first iterative prediction process, the first speech token is first predicted based on the input sequence; in the second iterative prediction process, the second speech token is predicted based on the input sequence and the first speech token; and so on, until all the speech units of the corresponding text are completed, and then the synthesized speech of the text is generated based on all the speech tokens. Due to the large parameters and large scale of the large language model, each iterative prediction consumes a lot of time, resulting in a long time and low efficiency for the overall speech synthesis. In addition, because the granularity of speech tokens is finer than that of text tokens, the large language model requires more iterative predictions, which further amplifies the disadvantage of the slow speech synthesis speed based on the large language model.

[0046] In view of this, an embodiment of the present application proposes a speech synthesis method that adopts the technical idea of ​​speculative sampling and adopts a speech synthesis scheme that uses both primary and secondary models. Specifically, a lightweight model with fewer parameters is first used to perform speech conversion on the first text based on a first sequence of text units containing the first text to obtain a speech unit sequence. Then, a heavyweight model with more parameters is used to verify the speech unit sequence with the help of the first sequence, and reliable speech units are determined and adopted. Furthermore, starting from the speech units that have not been adopted, the above operation is repeated until all the speech units corresponding to the first text are obtained, and the synthesized speech of the text is obtained based on these speech units.

[0047] Since the parameters of the lightweight model are much smaller than those of the heavyweight model, the prediction time of the lightweight model is much shorter than that of the heavyweight model; secondly, since the model uses the obtained speech units to predict the next speech unit and uses matrix calculation, based on the speech unit sequence output by the existing lightweight model, the heavyweight model can complete the verification of the speech unit sequence output by the lightweight model after one prediction, which is significantly reduced compared to the traditional multi-step prediction scheme. Moreover, although the reliability of the lightweight model is not as good as that of the heavyweight model, the quality of the verified speech units can be ensured after verification by the heavyweight model. It can be seen that the speech synthesis method proposed in the embodiment of the present application significantly improves the efficiency of speech synthesis while ensuring the accuracy of speech synthesis, and can meet the needs of real-time speech synthesis.

[0048] It should be understood that the speech synthesis method provided in the embodiments of the present application can be performed by an electronic device. The electronic device herein may include a terminal device, such as a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device, smart home appliance, smart watch, vehicle-mounted terminal, aircraft, etc.; or the electronic device may also include a server, such as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0049] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0050] Please refer to Figure 1 , is a flow chart of a speech synthesis method provided in one embodiment of the present application, the method comprising the following steps:

[0051] S102: Perform speech conversion on the first text based on the first sequence using a first model to obtain a first speech unit sequence.

[0052] The first sequence represents the relationship between text units in the first text, and is used as a guide or basis for speech conversion to convert the first text into speech.

[0053] As an example, the first sequence includes text units of the first text. A text unit, also known as a text token, is an element in the first text that can be considered as a separate unit. Depending on the granularity, a text unit can be a word, a phrase, a character, etc., which is not limited in this embodiment of the present application. The above-mentioned multiple text units can be obtained by segmenting the first text.

[0054] As another example, the first sequence may also include at least one of the following information: acoustic features such as the speaker's timbre and emotion, the start identifier and end identifier of the text unit, the start identifier of the speech unit, etc. Among them, the acoustic features of the speaker are used to make the speech synthesized for the first text more natural and closer to the speech characteristics of a real person. The start identifier of the text unit is used to clearly indicate the first text unit of the first text, and the end identifier of the unit is used to clearly indicate the last text unit of the first text. It can be seen that the start identifier and end identifier of the text unit help the model to better understand the context of each information in the first sequence when performing speech conversion on the first text, thereby improving the accuracy of speech conversion. The start identifier of the speech unit is used to clarify the beginning of the speech unit sequence to be generated, providing the model with a fixed starting point, which helps the model to perform speech conversion more accurately.

[0055] The first speech unit sequence includes multiple first speech units. In the embodiment of the present application, the speech unit is also called a speech token, which is an intermediate representation in the process of converting a text unit into a speech signal. It is the result of discretizing a continuous speech waveform or speech feature into a finite vocabulary or symbol list. As a bridge between text and speech, the speech token enables the text to be processed and understood more efficiently in the speech synthesis process and converted into corresponding synthesized speech output. It is worth noting that due to the difference in the bit rate of the text and the bit rate of the speech signal, in actual applications, one text unit may correspond to multiple first speech units. Therefore, the speech conversion process based on the first model is performed on data of non-equivalent resolution, which is something that models in the NLP field, especially large language models in the NLP field, do not need to consider.

[0056] In the embodiment of the present application, both the first model and the second model can be large language models, such as a large language model based on an AR architecture. The difference between the two is that the first model is a lightweight large language model, and the second model is a large language model of larger magnitude, that is, the parameters of the first model are less than the parameters of the second model. Based on this, the time it takes for the first model to perform speech conversion on the first text based on the first sequence is much shorter than the speech conversion time of the second model, which is conducive to improving the speech synthesis efficiency of the first text.

[0057] In the embodiment of the present application, the above S102 can be implemented in various appropriate ways.

[0058] In one implementation, the first sequence is obtained by concatenating the text identifier of the first text and the first start identifier, where the first start identifier refers to the start identifier of the speech unit described above. S102 includes the following steps: performing multiple sequence analysis on the first sequence using the first model to obtain multiple first speech units, and generating a first speech unit sequence based on the multiple first speech units. In the first prediction process, the first model predicts the first first speech unit after the first start identifier based on the first sequence; and in the kth prediction process, the first model predicts the kth first speech unit based on the first sequence and the first k-1 first speech units, where k is an integer greater than 1.

[0059] For example, suppose the first sequence is [spk][text begin][ttoken1,ttoken2,… ,ttokenN][text end][mel begin], and the first model makes five predictions for the first sequence. [spk] represents the speaker's acoustic features, [text begin] represents the start marker of a text unit, [ttoken1,ttoken2,… ,ttokenN] represents the text unit of the first text, [text end] represents the end marker of the text unit, and [mel begin] represents the first start marker.

[0060] In the first prediction process, the first sequence is input into the first model to obtain the first first speech unit mtoken1 after [mel begin].

[0061] In the second prediction process, the first sequence and mtoken1 are concatenated and input into the first model to obtain the second first speech unit mtoken2.

[0062] Similarly, the third prediction results in the third first speech unit mtoken3, the fourth prediction results in the fourth first speech unit mtoken4, and the fifth prediction results in the fifth first speech unit mtoken5. These first speech units are concatenated to obtain the first speech unit sequence [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5].

[0063] The above implementation is applicable to the case where the structure of the first model is similar to that of the second model, for example, the first model is obtained by simplifying the structure of the second model.

[0064] In another implementation, the first sequence is obtained by concatenating the text identifier and the first start identifier of the first text. Before the above S102, the method provided in the embodiment of the present application further includes: predicting the first second speech unit after the first start identifier based on the first sequence through the second model. In this case, the above S102 includes the following steps: predicting the first sequence multiple times through the first model to obtain multiple first speech units, and generating a first speech unit sequence based on the multiple first speech units. In the first prediction process, the first model predicts the first first speech unit after the first second speech unit based on the first sequence and the first second speech unit; in the kth prediction process, the first model predicts the kth first speech unit based on the first sequence, the first second speech unit and the first k-1 first speech units, where k is an integer greater than 1.

[0065] For example, still taking the first sequence mentioned above as an example, assume that the first model makes 5 predictions for the first sequence.

[0066] First, the first sequence is input into the second model to obtain the first second speech unit mtoken1' after [mel begin]. Then, the first language model performs multiple predictions based on the first sequence and mtoken1'. The specific process is as follows:

[0067] In the first prediction process, the first sequence and mtoken1′ are input into the first model to obtain the first first speech unit mtoken1 after mtoken1′.

[0068] In the second prediction process, the first sequence, mtoken1' and mtoken1 are concatenated and input into the first model to obtain the second first speech unit mtoken2.

[0069] Similarly, the third prediction results in the third first speech unit mtoken3, the fourth prediction results in the fourth first speech unit mtoken4, and the fifth prediction results in the fifth first speech unit mtoken5. These first speech units are concatenated to obtain the first speech unit sequence [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5].

[0070] Since the speech synthesis process is highly sensitive to speech units, if the quality of a speech unit is poor, the performance of the final synthesized speech will be poor, affecting user perception. In addition, the model predicts the next speech unit based on the existing speech unit. If the quality of the first speech unit is poor, the quality of all subsequent speech units will be affected. To this end, in the above implementation method, the first second speech unit is first predicted by the second model with high prediction accuracy, and then the first model predicts multiple subsequent first speech units based on the first second speech unit. This can ensure the accuracy of the first speech unit sequence and provide reliable data support for the subsequent generation of high-quality synthesized speech.

[0071] The above implementation is applicable to the case where the first model and the second model have significant structural differences.

[0072] The above shows some implementations of S102. Of course, it should be understood that the above S102 can also be implemented in other ways, which are not limited in the present embodiment.

[0073] In the embodiment of the present application, the number of times the first model predicts the first input unit can be set according to actual needs, and the embodiment of the present application does not limit this. This number can also be called the size of the speculation window.

[0074] In one implementation, the number of predictions for the first input unit using the first model is not limited until the first speech unit sequence includes a target first speech unit indicating a first end marker, wherein the first end marker is used to clearly indicate the end of the first speech unit.

[0075] In another implementation, to better balance the accuracy and efficiency of speech conversion performed by the first model, the number of predictions performed on the first input unit by the first model can be determined based on the structural similarity between the first model and the second model. For example, if the first model and the second model are structurally similar, the number of predictions can be set to a larger value, such as 5; if the first model and the second model are structurally different, the number of predictions can be set to a smaller value, such as 2.

[0076] In another implementation, the number of times can be determined based on the training effect of the first model. For example, if the first model has a large amount of training data, is fully trained, and has a good training effect, the number of times can be set to a larger value; otherwise, the number of times can be set to a smaller value.

[0077] S104 , performing speech conversion on the first text based on the first speech unit sequence and the first sequence using a second model to obtain a second speech unit sequence.

[0078] Typically, the second model performs speech conversion on the first text directly based on the first speech unit sequence, requiring multiple iterative predictions. Since the second model is a larger model with more parameters than the first model, the required conversion time is longer. In S104 above, since the model uses the obtained speech units to predict the next speech unit and uses matrix calculations, based on the first speech unit sequence already obtained by the lightweight model, the second model can achieve the same multiple prediction effect as the first model through a single prediction, thereby obtaining the second speech unit sequence.

[0079] In one implementation, the above S104 includes the following steps: concatenating the first sequence and the first speech unit sequence to obtain a second sequence; predicting the second sequence through a second model to obtain a first output sequence, the first output sequence including: the next fourth speech unit of the first start identifier, the next fourth speech unit of each first speech unit in the first speech unit sequence; sampling the first output sequence to obtain multiple second speech units after the first start identifier; and generating a second speech unit sequence based on the multiple second speech units.

[0080] For example, assuming the first sequence is [spk][text begin][ttoken1,ttoken2,… ,ttokenN][text end][mel begin], and the first speech unit sequence is [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5], concatenate the two to obtain the second sequence: [spk][text begin][ttoken1,ttoken2,… ,ttokenN][text end][mel begin] [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5].

[0081] The second sequence is input into the second model, and the second model predicts the probability distribution of the next unit of each information in the second sequence, and the following first output sequence is obtained:

[0082] [NXT spk][NXT text begin][NXT ttoken1] …[NXT mel begin][NXT mtoken1][NXT mtoken2],…,[NXT mtoken5]. Wherein, [NXT spk] indicates the next unit predicted for [spk]; [NXT text begin] indicates the next unit predicted for [text begin]; [NXT ttoken1] indicates the next unit predicted for [ttoken1]; [NXT mel begin] indicates the next unit predicted for [mel begin], also known as the next fourth phonetic unit of the first start marker; [NXT mtoken1] indicates the next unit predicted for [mtoken1], also known as the next fourth phonetic unit of [mtoken1]; [NXT mtoken2] indicates the next unit predicted for [mtoken2], also known as the next fourth phonetic unit of [mtoken2]; and [NXT mtoken5] indicates the next unit predicted for [mtoken5], also known as the next phonetic unit of [mtoken5].

[0083] Further, [NXT mel begin] is sampled to obtain the first second speech unit [mtoken1'] after [NXT mel begin]; [NXT mtoken1] is sampled to obtain the second second speech unit [mtoken2'] after [NXT mel begin]; and so on, 5 second speech units are obtained, and based on these second speech units, a second speech unit sequence [mtoken1'][mtoken2'][mtoken3'][mtoken4'][mtoken5'] is generated.

[0084] It can be seen that the second model can achieve multiple prediction effects of the first model after one prediction, and obtain the second speech unit sequence.

[0085] In another implementation, if the second model is used to first predict the first second speech unit, and then the first model is used to predict multiple first speech units after the second speech unit, the above S104 includes the following steps: splicing the first sequence, the first second speech unit and the first speech unit sequence to obtain a second sequence; outputting the second sequence through the second model to obtain a first output sequence, the first output sequence including: the next fourth speech unit of the first start identifier, the next fourth speech unit of each first speech unit in the first speech unit sequence; sampling the first output sequence to obtain multiple second speech units after the first start identifier; generating a second speech unit sequence based on the multiple second speech units.

[0086] Since the specific process of the above implementation is similar to that of the first implementation, it will not be described in detail. It is worth noting that in this implementation, when sampling multiple second speech units, the first second speech unit that has been generated is not included in the sampling.

[0087] The embodiment of the present application shows some implementations of the above S104. Of course, it should be understood that the above S104 can also be implemented in other ways, which are not limited in the embodiment of the present application.

[0088] S106: Generate a third speech unit sequence based on the matching result between the first speech unit sequence and the second speech unit sequence.

[0089] Matching the first speech unit sequence with the second speech unit sequence is to verify the first speech unit. The third speech unit sequence includes at least one third speech unit, which is a verified speech unit that can be used to synthesize the speech of the first text. Specifically, for each first speech unit in the first speech unit sequence, the first speech unit can be matched with the second speech unit at the same position in the second speech unit sequence. If the two match, for example, the similarity between the two is greater than a similarity threshold, then the first speech unit is determined to have passed verification and is used as the third speech unit.

[0090] In the embodiment of the present application, the above S106 can be implemented in various appropriate ways.

[0091] In one implementation, based on the matching results between the first speech unit sequence and the second speech unit sequence, it is determined whether a target first speech unit exists in the first speech unit sequence, wherein the target first speech unit is the first first speech unit that does not match the second speech unit at the same position in the second speech unit sequence; if the target first speech unit does not exist in the first speech unit sequence, the first speech unit sequence and the fourth speech unit next to the last speech unit in the first speech unit sequence are spliced ​​as the third speech unit to obtain a third speech unit sequence; if the target first speech unit exists in the first speech unit sequence, the first speech unit before the target first speech unit and the target second speech unit are spliced ​​as the third speech unit to obtain a third speech unit sequence, wherein the target second speech unit is the second speech unit in the second speech unit sequence with the same position as the target first speech unit.

[0092] For example, assuming that the first speech unit sequence is [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5], the second speech unit sequence is [mtoken1'][mtoken2'][mtoken3'][mtoken4'][mtoken5'], the next fourth speech unit after [mtoken5'] is [NXT mtoken5].

[0093] In the optimal case, each first phonetic unit in the first phonetic unit sequence matches the second phonetic unit at the same position in the second phonetic unit. Each first phonetic unit and [NXT mtoken5] are then used as the third phonetic unit, resulting in the following third phonetic unit sequence: [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5][NXTmtoken5]. In this case, the second model obtains six valid third phonetic units with only one prediction. Furthermore, in the next speculative sampling process, the first model predicts the seventh valid third phonetic unit and subsequent third phonetic units based on the third phonetic unit sequence.

[0094] In the suboptimal case, the third first speech unit [mtoken3] in the first speech unit sequence is the first first speech unit that does not match the third second speech unit [mtoken3'] in the second speech unit sequence. The first two first speech units and the third second speech unit are then concatenated as the third speech unit, resulting in the third speech unit sequence: [mtoken1][mtoken2][mtoken3']. In this case, the second model only makes one prediction to obtain three valid third speech units. Furthermore, in the next speculative sampling process, the first model predicts the fourth valid third speech unit and subsequent third speech units based on the third speech unit sequence.

[0095] In the worst case, the first and second speech unit sequences do not match, and the first second speech unit is used as the third speech unit, resulting in the third speech unit sequence: [mtoken1']. In this case, the second model predicts a valid third speech unit in a single pass. Furthermore, during the next speculative sampling process, the first model predicts the second valid third speech unit and subsequent third speech units based on the third speech unit sequence.

[0096] In another implementation, based on the matching result between the first speech unit sequence and the second speech unit sequence, it is determined whether the target first speech unit exists in the first speech unit sequence; if the target first speech unit does not exist in the first speech unit sequence, the first speech unit sequence is used as the third speech unit sequence; if the target first speech unit exists in the first speech unit sequence, the first speech unit before the target first speech unit and the target second speech unit are spliced ​​as the third speech unit to obtain the third speech unit sequence.

[0097] The specific process of this implementation is similar to that of the first implementation and will not be repeated here.

[0098] The embodiment of the present application shows some implementations of the above S106. Of course, it should be understood that the above S106 can also be implemented in other ways, which are not limited in the embodiment of the present application.

[0099] S108: Generate synthesized speech of the first text based on the third speech unit sequence.

[0100] Since the model stops speech conversion based on the first end marker (i.e., the end marker of the speech unit sequence), in order to accurately control the length and quality of the third speech unit sequence used to synthesize the first text, the above S108 includes the following steps: determining whether the third speech unit sequence contains the target third speech unit representing the first end marker; if the third speech unit sequence contains the target third speech unit, concatenating the first sequence with the third speech unit sequence to obtain a new first sequence, and repeatedly executing the above S102 to S108 until the third speech unit sequence contains the target third speech unit; if the third speech unit sequence contains the target third speech unit, generating the synthesized speech of the first text based on the third speech unit sequence.

[0101] For example, assuming the first model predicts five times, after the first round of speech conversion, a third speech unit sequence is obtained: [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5][NXT mtoken5]. Furthermore, the third speech unit sequence is concatenated after the first sequence to obtain a new first sequence. Steps S102 to S108 are repeated until the obtained third speech unit sequence contains the target third speech unit [mel end]. Furthermore, post-processing is performed based on all third speech unit sequences to convert them into synthesized speech for the first text. At this point, speech synthesis for the first text is completed.

[0102] The speech synthesis method provided in the embodiment of the present application adopts the technical idea of ​​speculative sampling and adopts a speech synthesis scheme that uses both main and auxiliary models. Specifically, a lightweight model with fewer parameters is first used to perform speech conversion on the first text based on a first sequence of text units containing the first text to obtain a first speech unit sequence. Then, a heavyweight model with more parameters is used to perform a speech conversion on the first text with the help of the first sequence and the first speech unit sequence to obtain a second speech unit sequence. The first speech unit sequence is verified using the second speech unit sequence, and reliable speech units are determined and adopted. Furthermore, starting from the speech units that have not been adopted, the above operation is repeated until all speech units corresponding to the first text are obtained, and the synthesized speech of the text is obtained based on these speech units.

[0103] Since the parameters of the lightweight model are much smaller than those of the heavyweight model, the prediction time of the lightweight model is much shorter than that of the heavyweight model; secondly, since the model uses the obtained speech units to predict the next speech unit and uses matrix calculation, based on the speech unit sequence output by the existing lightweight model, the heavyweight model can complete the verification of the speech unit sequence output by the lightweight model after one prediction, which is significantly reduced compared to the traditional multi-step prediction scheme. Moreover, although the reliability of the lightweight model is not as good as that of the heavyweight model, the quality of the verified speech units can be ensured after verification by the heavyweight model. It can be seen that the speech synthesis method proposed in the embodiment of the present application significantly improves the efficiency of speech synthesis while ensuring the accuracy of speech synthesis, and can meet the needs of real-time speech synthesis.

[0104] The present application also provides a method for training a first model. Figure 2 , is a flow chart of a first model training method provided in one embodiment of the present application, the method comprising the following steps:

[0105] S202: Obtain a third model, and obtain a first sample sequence used to train the second model and a first sample speech unit sequence corresponding to the first sample sequence.

[0106] The first sample sequence includes text units of the second text. The first sample speech unit sequence includes multiple first sample speech units used to obtain the synthesized speech of the second text. The first sample sequence and its corresponding first sample speech unit sequence are training data for the second model. For example, Figure 3 A schematic diagram of training data for a second model is shown. In the training data, speaker represents the acoustic features of the speaker, text begin represents the start identifier of the text unit of the second text, text end represents the end identifier of the text unit of the second text, and the text units of the second text are located between the two. mel begin is the second start identifier, which is used to indicate the position of the first first sample speech unit, and mel end is the second end identifier, which is used to indicate the position of the last sample speech unit. The text units of the second text are located between the two. The sequence consisting of text begin, the text unit of the second text, text end, and mel begin is a first sample sequence, and the sequence consisting of multiple first sample speech units and mel end is a first sample speech unit sequence.

[0107] The third model refers to a lightweight large language model to be trained. The first model can have various appropriate structures, which are not limited in the present embodiment.

[0108] In one embodiment, the third model has a similar structure to the second model. Figure 4 As shown, the second model includes n second conversion layers, where n is an integer greater than 1. The second conversion layer can use a transformer. Optionally, the second model can also include an embedding layer and a prediction layer. The embedding layer is used to convert the sequence of the second model into a corresponding vector representation, and the n second conversion layers are used to determine the relationship between each information in the sequence based on the vector representation and the attention mechanism to obtain a latent vector sequence; the prediction layer is used to obtain a corresponding speech unit sequence based on the latent vector sequence.

[0109] The third model can be obtained by cutting the structure of the second model. Specifically, Figure 5 As shown, the first model includes m first conversion layers, where m is an integer greater than 1 and m is less than n. The value of m can be set according to actual needs, for example, m=3. Based on the parameters of the 1st second conversion layer, the parameters of the 1st first conversion layer are initialized; based on the parameters of the nth second conversion layer, the parameters of the mth first conversion layer are initialized; based on the parameters of the jth second conversion layer, the parameters of the ith first conversion layer are initialized, wherein i and j are both integers greater than 1, and i<m, j<n. Exemplarily, the parameters of the 6th second conversion layer are used to initialize the parameters of the 3rd first conversion layer. In addition, the parameters of the embedding layer of the second model can also be used to initialize the parameters of the embedding layer of the first model.

[0110] In practical applications, each first conversion layer may be initialized by using a normal distribution random initialization method.

[0111] Since the structure of the third model is similar to that of the second model, initializing the parameters of the third model using the parameters of the second model can improve the prediction accuracy of the third model. In addition, since the first conversion layer is closer to the text, it has a stronger ability to learn the contextual relationships between text units. Initializing the parameters of the first conversion layer using the parameters of the first second conversion layer can ensure that the third model also has the ability to accurately learn the contextual relationships between text units. The last conversion layer mainly learns the association between text units and speech units, which greatly affects the prediction accuracy of speech units. Initializing the parameters of the last first conversion layer using the parameters of the last second conversion layer can ensure that the third model also has the ability to accurately learn the association between text units and speech units. The conversion layer in the middle plays a transition role. Initializing the parameters of the first conversion layer in the middle using the parameters of the second conversion layer can improve the transition ability between the various conversion layers in the third model.

[0112] In another embodiment, the third model has a different structure than the second model. Figure 6 As shown in the figure, the third model adopts a simplified sequence prediction model, which includes a convolutional layer, a first long short-term memory network (LSTM), a first normalization and pooling layer, a second long short-term memory network, a second normalization and pooling layer, and a classification layer.

[0113] The convolutional layer is used for channel compression. Since the inner layer latent variables of the third model have high dimensions, which is not conducive to sequence prediction learning, the convolutional layer can compress the dimensions of the latent variables.

[0114] The first LSTM, the first normalization and pooling layer, the second LSTM, and the second normalization and pooling layer are used for context learning. The first and second LSTMs primarily learn long-term and short-term dependencies between sequences. The output of the first LSTM is regularized and pooled by the first normalization and pooling layer and serves as the input to the second LSTM. The output of the second LSTM is regularized and pooled by the second normalization and pooling layer and serves as the input to the classification layer. The classification layer, which includes a softmax activation layer and a linear layer, predicts speech units based on the output of the second LSTM.

[0115] S204 , performing speech conversion on the second text based on the first sample sequence using a third model to obtain a second sample speech unit sequence.

[0116] The second sample speech unit sequence includes a plurality of second sample speech units arranged in sequence.

[0117] In one implementation, if the structure of the third model is similar to that of the second model, for example Figure 5 The third model shown in FIG. 1 predicts the first sample sequence multiple times using the third model until the obtained second sample speech unit sequence contains a second sample speech unit indicating the second end marker. In the first prediction process, the third model predicts the first second sample speech unit based on the first sample sequence; in the kth prediction process, the third model predicts the kth second sample speech unit based on the first sample sequence and the first k-1 second sample speech units, where k is an integer greater than 1.

[0118] In another implementation, if the structure of the third model is significantly different from that of the second model, for example Figure 6If the third model is shown, the first sample speech unit sequence also includes a second end marker after the last first sample speech unit. In this case, the above S204 includes: concatenating the first sample sequence and the first sample speech unit sequence and inputting them into the third model for prediction; obtaining a second output sequence of the nth second conversion layer, the second output sequence including: a second start marker and multiple latent vectors after the second start marker; concatenating the second start marker and the multiple latent vectors to obtain a second sample sequence; and performing speech conversion on the second text based on the second sample sequence using the third model to obtain a second sample speech unit sequence.

[0119] For example, Figure 7 As shown, the first sample sequence includes: spk, text begin, a text unit of the second text, text end, and mel begin (second start marker). The first sample speech unit sequence includes: multiple first sample speech units and mel end. The first sample sequence and the first sample speech unit sequence are concatenated to obtain a sequence with a total length of L. This sequence is input into the second model for prediction, and the second output sequence of the nth second conversion layer is obtained. The total length of the second output sequence is L, and a subsequence from mel begin to mel end is intercepted to obtain a second sample sequence of length subL. Furthermore, the second sample sequence is input into the third model for multiple predictions to obtain a second sample speech unit sequence.

[0120] Because the second model has a high prediction accuracy, the second output sequence predicted from the last second conversion layer of the second model for the first sample sequence and the first sample speech unit sequence has a high accuracy rate. This implicitly implies the association between the text units of the second text and the first sample speech units, which plays an auxiliary role in the training of the third model and helps improve the training effect of the third model. In addition, adding a second end marker to the first sample speech unit sequence and a second start marker to the second sample sequence can achieve alignment of the first sample speech unit sequence with the second sample sequence, allowing the third model to better learn the association between text units and speech units, further improving the training effect of the third model.

[0121] S206: Based on the matching result between the first sample speech unit sequence and the second sample speech unit sequence, adjust the parameters of the third model to obtain the first model.

[0122] Specifically, based on the difference between the first sample speech unit sequence and the second sample speech unit sequence, the loss of the third model is determined; then, the parameters of the third model are adjusted using a back propagation algorithm, a gradient descent algorithm, etc.

[0123] The above steps S202 to S206 represent only one training process for the third model. In practical applications, the third model may be trained multiple times until a training stop condition is met. The training stop condition may be set as needed, for example, when the number of training times reaches a threshold, when the third model converges, or when the loss of the third model is less than a loss threshold, etc., and is not limited in this embodiment of the present application.

[0124] The training method for the first model provided in the embodiment of the present application abandons the traditional distillation technology or bypass training technology, and instead uses the training data of the second model to train the third model to obtain the first model. This can reduce the loss generated by the first model in the prediction process, improve the prediction accuracy of the first model, and thus improve the accuracy of speech synthesis based on the first model.

[0125] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0126] Based on the same inventive concept, the present application also provides a speech synthesis device. Figure 8 , is a structural diagram of a speech synthesis device 800 provided in an embodiment of the present application, wherein the device 800 includes: a first generation module 810, a second generation module 820, a third generation module 830 and a speech synthesis module 840.

[0127] The first generating module 810 is configured to perform speech conversion on the first text based on a first sequence using a first model to obtain a first speech unit sequence, where the first sequence includes text units of the first text.

[0128] The second generation module 820 is used to perform speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence; the parameters of the first model are less than the parameters of the second model.

[0129] The third generating module 830 is configured to generate a third speech unit sequence based on the matching result between the first speech unit sequence and the second speech unit sequence.

[0130] The speech synthesis module 840 is configured to generate synthesized speech of the first text based on the third speech unit sequence.

[0131] In another embodiment, the first sequence is obtained by concatenating the text unit of the first text and the first start marker;

[0132] The second generation module is further configured to predict, by using the second model, a first second speech unit following the first start identifier based on the first sequence;

[0133] The first generating module is used for:

[0134] Predicting the first sequence multiple times using the first model to obtain multiple first speech units, and generating the first speech unit sequence based on the multiple first speech units;

[0135] Wherein, in the first prediction process, the first model predicts the first first speech unit after the first second speech unit based on the first sequence and the first second speech unit;

[0136] In the kth prediction process, the first model predicts the kth first speech unit based on the first sequence, the first second speech unit and the first k-1 first speech units, where k is an integer greater than 1.

[0137] In another embodiment, the first sequence further includes a first start marker;

[0138] The second generating module is used for:

[0139] concatenating the first sequence and the first speech unit sequence to obtain a second sequence;

[0140] Predicting the second sequence using the second model to obtain a first output sequence, the first output sequence including: the fourth speech unit next to the first start identifier, and the fourth speech unit next to each first speech unit in the first speech unit sequence;

[0141] Sampling the first output sequence to obtain a plurality of second speech units following the first start identifier;

[0142] Based on the plurality of second speech units, a second speech unit sequence is generated.

[0143] In another embodiment, the third generation module is used to:

[0144] Based on the matching result, determining whether there is a target first speech unit in the first speech unit sequence, the target first speech unit being the first first speech unit that does not match the second speech unit at the same position in the second speech unit sequence;

[0145] If the target first speech unit does not exist in the first speech unit sequence, concatenating the first speech unit sequence and the fourth speech unit next to the last speech unit in the first speech unit sequence as a third speech unit to obtain a third speech unit sequence;

[0146] If the target first speech unit exists in the first speech unit sequence, the first speech unit before the target first speech unit and the target second speech unit are spliced ​​as the third speech unit to obtain a third speech unit sequence, wherein the target second speech unit is the second speech unit in the second speech unit sequence with the same position as the target first speech unit.

[0147] In another embodiment, the speech synthesis module is used to:

[0148] determining whether the third speech unit sequence includes a target third speech unit indicating a first end marker;

[0149] If the third speech unit sequence includes the target third speech unit, concatenating the first sequence with the third speech unit sequence to obtain a new first sequence, and calling the first generation module, the second generation module, and the third generation module to repeatedly perform the steps of performing speech conversion on the first text based on the first sequence through the first model to determining whether the third speech unit sequence includes the target third speech unit indicating the first end marker, until the third speech unit sequence includes the target third speech unit;

[0150] If the third speech unit sequence includes the target third speech unit, synthesized speech of the first text is generated based on the third speech unit sequence.

[0151] In another embodiment, the first model is obtained by:

[0152] Obtaining a third model, and obtaining a first sample sequence used to train the second model and a first sample speech unit sequence corresponding to the first sample sequence, wherein the first sample sequence includes text units of the second text, and the first sample speech unit sequence includes multiple first sample speech units used to obtain synthesized speech of the second text;

[0153] performing speech conversion on the second text based on the first sample sequence using the third model to obtain a second sample speech unit sequence, where the second sample speech unit sequence includes a plurality of second sample speech units arranged in sequence;

[0154] Based on the matching result between the first sample speech unit sequence and the second sample speech unit sequence, the parameters of the third model are adjusted to obtain the first model.

[0155] In another embodiment, the third model includes m first conversion layers, the second model includes n second conversion layers, m is an integer greater than 1, and m is less than n;

[0156] Before performing speech conversion on the second text based on the first sample sequence using the third model to obtain a second sample speech unit sequence, the method further includes:

[0157] Initializing parameters of the first first conversion layer based on parameters of the first second conversion layer;

[0158] Initializing parameters of the mth first transformation layer based on parameters of the nth second transformation layer;

[0159] Initialize the parameters of the i-th first transformation layer based on the parameters of the j-th second transformation layer, wherein i and j are both integers greater than 1, i<m, j<n.

[0160] In another embodiment, the second model includes n second conversion layers, where n is an integer greater than 1; the first sample speech unit sequence further includes a second end marker after the last first sample speech unit;

[0161] The method of performing speech conversion on the second text based on the first sample sequence by the third model to obtain a second sample speech unit sequence includes:

[0162] splicing the first sample sequence and the first sample speech unit sequence and inputting them into the second model for prediction, and

[0163] Obtain a second output sequence of the nth second transformation layer, the second output sequence comprising: a second start identifier, and a plurality of latent vectors following the second start identifier;

[0164] concatenating the second start identifier and the multiple latent vectors to obtain a second sample sequence;

[0165] The second text is converted into speech based on the second sample sequence using the third model to obtain a second sample speech unit sequence.

[0166] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 9At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0167] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0168] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0169] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a speech synthesis device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0170] Performing speech conversion on a first text based on a first sequence using a first model to obtain a first speech unit sequence, where the first sequence includes text units of the first text;

[0171] performing speech conversion on the first text based on the first speech unit sequence and the first sequence using a second model to obtain a second speech unit sequence; wherein the parameters of the first model are less than the parameters of the second model;

[0172] generating a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence;

[0173] Based on the third speech unit sequence, synthesized speech of the first text is generated.

[0174] The above application Figure 1The methods performed by the speech synthesis device disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits within the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0175] The electronic device may also perform Figure 1 Method, and realize speech synthesis device in Figure 1 、 Figure 2 , the functions of the embodiments shown in the figure will not be described in detail in the embodiments of the present application.

[0176] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0177] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including multiple application programs, can enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations:

[0178] Performing speech conversion on a first text based on a first sequence using a first model to obtain a first speech unit sequence, where the first sequence includes text units of the first text;

[0179] performing speech conversion on the first text based on the first speech unit sequence and the first sequence using a second model to obtain a second speech unit sequence; wherein the parameters of the first model are less than the parameters of the second model;

[0180] generating a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence;

[0181] Based on the third speech unit sequence, synthesized speech of the first text is generated.

[0182] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps in the speech synthesis method provided in the embodiment of the present application.

[0183] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0184] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0185] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0186] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0187] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A speech synthesis method, characterized in that: include: Performing speech conversion on a first text based on a first sequence using a first model to obtain a first speech unit sequence, where the first sequence includes text units of the first text; performing speech conversion on the first text based on the first speech unit sequence and the first sequence using a second model to obtain a second speech unit sequence; wherein the parameters of the first model are less than the parameters of the second model; generating a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence; generating a synthesized speech of the first text based on the third speech unit sequence; Among them, the first model is obtained by adjusting the parameters of the third model based on the first sample speech unit sequence and the second sample speech unit sequence, the first sample speech unit sequence corresponds to the first sample sequence used to train the second model, the first sample sequence includes text units of the second text, and the second sample speech unit sequence is obtained by the third model performing speech conversion on the second text based on the first sample sequence; if the third model is obtained by trimming the structure of the second model, then before obtaining the second sample speech unit sequence, the parameters of the third model are also initialized based on the parameters of the second model.

2. The method according to claim 1, characterized in that The first sequence is obtained by concatenating the text unit of the first text and the first start marker; Before performing speech conversion on the first text based on the first sequence by using the first model to obtain the first speech unit sequence, the method further includes: predicting, by the second model, a first second speech unit after the first start identifier based on the first sequence; The performing speech conversion on the first text based on the first sequence by using the first model to obtain a first speech unit sequence includes: Predicting the first sequence multiple times using the first model to obtain multiple first speech units, and generating the first speech unit sequence based on the multiple first speech units; Wherein, in the first prediction process, the first model predicts the first first speech unit after the first second speech unit based on the first sequence and the first second speech unit; In the kth prediction process, the first model predicts the kth first speech unit based on the first sequence, the first second speech unit and the first k-1 first speech units, where k is an integer greater than 1.

3. The method according to claim 1, characterized in that The first sequence also includes a first start identifier; The step of performing speech conversion on the first text based on the first speech unit sequence and the first sequence by using a second model to obtain a second speech unit sequence includes: concatenating the first sequence and the first speech unit sequence to obtain a second sequence; Predicting the second sequence using the second model to obtain a first output sequence, the first output sequence including: the fourth speech unit next to the first start identifier, and the fourth speech unit next to each first speech unit in the first speech unit sequence; Sampling the first output sequence to obtain a plurality of second speech units following the first start identifier; Based on the plurality of second speech units, a second speech unit sequence is generated.

4. The method according to claim 3, characterized in that The generating a third speech unit sequence based on the matching result between the first speech unit sequence and the second speech unit sequence includes: Based on the matching result, determining whether there is a target first speech unit in the first speech unit sequence, the target first speech unit being the first first speech unit that does not match the second speech unit at the same position in the second speech unit sequence; If the target first speech unit does not exist in the first speech unit sequence, concatenating the first speech unit sequence and the fourth speech unit next to the last speech unit in the first speech unit sequence as a third speech unit to obtain a third speech unit sequence; If the target first speech unit exists in the first speech unit sequence, the first speech unit before the target first speech unit and the target second speech unit are spliced ​​as the third speech unit to obtain a third speech unit sequence, wherein the target second speech unit is the second speech unit in the second speech unit sequence with the same position as the target first speech unit.

5. The method according to claim 1, wherein Generating the synthesized speech of the first text based on the third speech unit sequence includes: determining whether the third speech unit sequence includes a target third speech unit indicating a first end marker; If the third speech unit sequence includes the target third speech unit, concatenating the first sequence with the third speech unit sequence to obtain a new first sequence, and repeatedly performing the steps of performing speech conversion on the first text based on the first sequence using the first model to determining whether the third speech unit sequence includes the target third speech unit indicating the first end marker, until the third speech unit sequence includes the target third speech unit; If the third speech unit sequence includes the target third speech unit, synthesized speech of the first text is generated based on the third speech unit sequence.

6. The method according to claim 1, characterized in that The third model is obtained by trimming the structure of the second model, the third model includes m first conversion layers, and the second model includes n second conversion layers, where m is an integer greater than 1 and less than n; Initializing parameters of the third model based on parameters of the second model includes: Initializing parameters of the first first conversion layer based on parameters of the first second conversion layer; Initializing parameters of the mth first transformation layer based on parameters of the nth second transformation layer; Initialize the parameters of the i-th first transformation layer based on the parameters of the j-th second transformation layer, wherein i and j are both integers greater than 1, i<m, j<n.

7. The method according to claim 1, characterized in that The second model includes n second conversion layers, where n is an integer greater than 1; if the third model has a different structure from the second model, the first sample speech unit sequence includes: a plurality of first sample speech units used to obtain the synthesized speech of the second text, and a second end marker after the last first sample speech unit; The second sample speech unit sequence is obtained by: splicing the first sample sequence and the first sample speech unit sequence and inputting them into the second model for prediction, and Obtain a second output sequence of the nth second transformation layer, the second output sequence comprising: a second start identifier, and a plurality of latent vectors following the second start identifier; concatenating the second start identifier and the multiple latent vectors to obtain a second sample sequence; The second text is converted into speech based on the second sample sequence using the third model to obtain a second sample speech unit sequence.

8. A speech synthesis device, characterized in that: include: a first generating module, configured to perform speech conversion on a first text based on a first sequence using a first model to obtain a first speech unit sequence, where the first sequence includes text units of the first text; a second generating module, configured to perform speech conversion on the first text based on the first speech unit sequence and the first sequence using a second model to obtain a second speech unit sequence; wherein the parameters of the first model are less than the parameters of the second model; a third generating module, configured to generate a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence; a speech synthesis module, configured to generate synthesized speech of the first text based on the third speech unit sequence; Among them, the first model is obtained by adjusting the parameters of the third model based on the first sample speech unit sequence and the second sample speech unit sequence. The first sample speech unit sequence corresponds to the first sample sequence used to train the second model. The first sample sequence includes text units of the second text. The second sample speech unit sequence is obtained by the third model performing speech conversion on the second text based on the first sample sequence. If the third model is obtained by trimming the structure of the second model, the parameters of the third model are also initialized based on the parameters of the second model before obtaining the second sample speech unit sequence.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the speech synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the speech synthesis method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the speech synthesis method according to any one of claims 1 to 7.

Images