Speech synthesis method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a speech synthesis method and device, electronic equipment, a storage medium and a program product, which are used for improving the speech synthesis efficiency based on a model and meeting real-time speech synthesis requirements. The speech synthesis method comprises the steps that speech conversion is carried out on a first text based on a first sequence through a first model, a first speech unit sequence is obtained, and the first sequence comprises text units of the first text; performing voice conversion on the first text based on the first voice unit sequence and the first sequence through a second model to obtain a second voice unit sequence; the parameters of the first model are less than those of the second model; generating a third voice unit sequence based on a matching result between the first voice unit sequence and the second voice unit sequence; and generating synthetic speech of the first text based on the third speech unit sequence.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method, device, electronic device, storage medium and program product. Background Art

[0002] Large language model (LLM) technology is in the ascendant, and various phenomenal products have emerged in the field of natural language processing (NLP), which can provide real-life online text conversation services. The main bottleneck of large language model technology is the huge computing power required. When reasoning with the model, a large number of floating-point calculations are required, and the longer the sequence to be generated, the greater the computing power required.

[0003] Text-to-speech (TTS) technology based on large language models, especially TTS technology based on large autoregressive (AR) language models, is also limited by computing power and difficult to apply to real-time speech generation services. Compared with large language models in the NLP field, the reasoning of TTS large language models takes longer, resulting in low speech synthesis efficiency.

[0004] Therefore, how to improve the efficiency of speech synthesis based on large language models and meet the needs of real-time speech synthesis has become a hot research topic. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a speech synthesis method, device, electronic device, storage medium and program product, which are used to improve the efficiency of model-based speech synthesis and meet the needs of real-time speech synthesis.

[0006] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions: In a first aspect, an embodiment of the present application provides a speech synthesis method, comprising: Performing speech conversion on the first text based on the first sequence by using a first model to obtain a first speech unit sequence, wherein the first sequence includes text units of the first text; performing speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence; the parameters of the first language model are less than the parameters of the second model; generating a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence; Based on the third speech unit sequence, synthesized speech of the first text is generated.

[0007] In a second aspect, an embodiment of the present application provides a speech synthesis device, including: A first generating module, configured to perform speech conversion on a first text based on a first sequence by using a first model to obtain a first speech unit sequence, wherein the first sequence includes text units of the first text; A second generating module is used to perform speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence; the parameters of the first model are less than the parameters of the second model; A third generating module, configured to generate a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence; A speech synthesis module is used to generate synthesized speech of the first text based on the third speech unit sequence.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the speech synthesis method provided in the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform the speech synthesis method provided in the second aspect.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps in the speech synthesis method of the first aspect.

[0011] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: The technical idea of ​​speculative sampling is adopted, and a speech synthesis scheme using both the main and auxiliary models is adopted. Specifically, a lightweight model with fewer parameters is first used to perform speech conversion on the first text based on the first sequence of text units containing the first text to obtain a first speech unit sequence, and then a heavyweight model with more parameters is used to perform a speech conversion on the first text with the help of the first sequence and the first speech unit sequence to obtain a second speech unit sequence, and the first speech unit sequence is used to verify the first speech unit sequence, and reliable speech units are determined and adopted; further, starting from the speech units that have not been adopted, the above operations are repeated until all speech units corresponding to the first text are obtained, and the synthesized speech of the text is obtained based on these speech units.

[0012] Since the parameters of the lightweight model are much smaller than those of the heavyweight model, the prediction time of the lightweight model is much shorter than that of the heavyweight model; secondly, since the large model uses the obtained speech units to predict the next speech unit and uses matrix calculation, based on the speech unit sequence output by the existing lightweight model, the heavyweight model can complete the verification of the speech unit sequence output by the lightweight model after one prediction, which significantly reduces the time consumption compared to the traditional multi-step prediction scheme. Moreover, although the reliability of the lightweight model is not as good as that of the heavyweight model, the quality of the verified speech unit can be ensured after verification by the heavyweight model. It can be seen that the speech synthesis method proposed in the embodiment of the present application significantly improves the speech synthesis efficiency while ensuring the accuracy of speech synthesis, and can meet the needs of real-time speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a speech synthesis method provided for one embodiment of the present application; Figure 2 A schematic flow chart of a first model training method provided for one embodiment of the present application; Figure 3 A schematic diagram of training data of a second model provided for one embodiment of the present application; Figure 4 A schematic diagram of the structure of a second model provided for an embodiment of the present application; Figure 5 A schematic diagram of the structure of a first model provided for an embodiment of the present application; Figure 6 A schematic structural diagram of a first model provided for another embodiment of the present application; Figure 7 A schematic diagram of a method for acquiring a second sample sequence provided by an embodiment of the present application; Figure 8 A schematic diagram of the structure of a speech synthesis device provided in one embodiment of the present application; Fig. 9 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0015] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0016] Explanation of relevant terms: Large language model: Generally, a large-scale language model is trained on a large computing platform using a large amount of training data. For large language models, self-supervised or semi-supervised training is generally used.

[0017] Autoregressive Large Language Model: A method for constructing LLM, which is the mainstream technical method of LLM. The characteristics of the autoregressive large language model are that it adopts a decoder-only architecture and a left-to-right unidirectional prediction training that conforms to the characteristics of natural language. During model inference, each inference step predicts only one unit (token). After the unit is predicted, it will be added to the end of the input sequence as the prediction input for the next unit.

[0018] Based on TTS large language model: A TTS system with large-scale parameters trained on a high-computing platform based on a large amount of voice data. Its characteristics are that it converts continuous voice signals into discrete voice units (tokens) through voice unitization technology, and then regards voice units and text units as equal units, thereby constructing an LLM large language model that generates voice units from text units in one language model.

[0019] Speculative Sampling: A technical idea is to find a way to train a small model with fewer parameters outside of LLM. The small model has lower accuracy than LLM, but it is faster. Therefore, the time taken to perform multi-step sampling with a small model may be shorter than the time taken to perform one-step sampling with LLM.

[0020] Transformer: It is the basis of the current large language model. Its multi-layer stacked model structure based on multi-head attention constitutes the main structure of the modern LLM.

[0021] As mentioned above, the TTS large language model is also limited by computing power and is difficult to apply to real-time speech generation services. Compared with the large language model in the NLP field, the inference of the TTS large language model takes longer, resulting in low speech synthesis efficiency. This is because, in general, the bit rate of the quantized speech signal is higher than the text bit rate. For example, taking the text "I like this color" as an example, assuming that one character is one code, the bit rate of the text is 7 text tokens per second, but for the speech of the text, assuming that the length of the speech is 1 second, according to the sampling rate of 16k, it is 16,000 sampling points. The general speech codec (Coder-Decoder, codec) technology can compress the speech to 30~50 tokens, and the maximum compression is about 20 tokens. Then the bit rate of the speech is 20 speech tokens per second, which is much higher than the bit rate of the text. In the TTS scenario, the speech token is exactly the target of the large language model generation. Therefore, for TTS technology based on the large language model, it is urgent to improve the speech synthesis speed to meet the real-time speech synthesis needs.

[0022] The applicant studied the traditional TTS technology based on the large language model and found that the main reason for the low efficiency of speech synthesis is that the large language model needs to go through multiple iterative predictions in the process of language synthesis based on the input sequence of text tokens containing text, that is, in the first iterative prediction process, the first speech token is first predicted based on the input sequence; in the second iterative prediction process, the second speech token is predicted based on the input sequence and the first speech token; and so on, until all the speech units of the corresponding text are completed, and then based on all the speech tokens, the synthesized speech of the text is generated. Due to the large parameters and large scale of the large language model, each iterative prediction will consume a lot of time, resulting in a long time and low efficiency for the overall speech synthesis. In addition, since the granularity of speech tokens is finer than that of text tokens, the large language model requires more iterative predictions, which further amplifies the disadvantage of slow speech synthesis based on the large language model.

[0023] In view of this, the embodiment of the present application proposes a speech synthesis method, which adopts the technical idea of ​​speculative sampling and adopts a speech synthesis scheme using both main and auxiliary models. Specifically, a lightweight model with fewer parameters is first used to perform speech conversion on the first text based on a first sequence of text units containing the first text to obtain a speech unit sequence, and then a heavyweight model with more parameters is used to verify the speech unit sequence with the help of the first sequence, and reliable speech units are determined and adopted; further, starting from the speech units that have not been adopted, the above operations are repeated until all speech units corresponding to the first text are obtained, and the synthesized speech of the text is obtained based on these speech units.

[0024] Since the parameters of the lightweight model are much smaller than those of the heavyweight model, the prediction time of the lightweight model is much shorter than that of the heavyweight model; secondly, since the model uses the obtained speech units to predict the next speech unit and uses matrix calculation, based on the speech unit sequence output by the existing lightweight model, the heavyweight model can complete the verification of the speech unit sequence output by the lightweight model after one prediction, which significantly reduces the time consumption compared to the traditional multi-step prediction scheme. Moreover, although the reliability of the lightweight model is not as good as that of the heavyweight model, the quality of the verified speech unit can be ensured after verification by the heavyweight model. It can be seen that the speech synthesis method proposed in the embodiment of the present application significantly improves the speech synthesis efficiency while ensuring the accuracy of speech synthesis, and can meet the needs of real-time speech synthesis.

[0025] It should be understood that the speech synthesis method provided in the embodiments of the present application can be performed by an electronic device. The electronic device referred to here may include a terminal device, such as a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart home appliance, a smart watch, a vehicle terminal, an aircraft, etc.; or, the electronic device may also include a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0026] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0027] Please refer to Figure 1 , is a flow chart of a speech synthesis method provided by an embodiment of the present application, the method comprising the following steps: S102, performing speech conversion on the first text based on the first sequence using a first model to obtain a first speech unit sequence.

[0028] The first sequence represents the relationship between text units in the first text, and is used as a guide or basis for speech conversion to convert the first text into speech.

[0029] As an example, the first sequence includes text units of the first text. A text unit is also called a text token, which is an element in the first text that can be regarded as a single unit. Depending on the granularity, a text unit can be a word, a phrase, a character, etc., which is not limited in the embodiments of the present application. The above multiple text units can be obtained by segmenting the first text.

[0030] As another example, the first sequence may also include at least one of the following information: acoustic features such as the speaker's timbre and emotion, the start mark and end mark of the text unit, the start mark of the speech unit, etc. Among them, the speaker's acoustic features are used to make the speech synthesized for the first text more natural and close to the speech characteristics of a real person. The start mark of the text unit is used to clearly indicate the first text unit of the first text, and the end mark of this unit is used to clearly indicate the last text unit of the first text. It can be seen that the start mark and end mark of the text unit help the model to better understand the context of each information in the first sequence when converting the first text into speech, thereby improving the accuracy of speech conversion. The start mark of the speech unit is used to clarify the beginning of the speech unit sequence to be generated, providing a fixed starting point for the model, which helps the model to perform speech conversion more accurately.

[0031] The first speech unit sequence includes multiple first speech units. In the embodiment of the present application, the speech unit is also called a speech token, which is an intermediate representation in the process of converting a text unit into a speech signal. It is the result of discretizing a continuous speech waveform or speech feature into a finite vocabulary or symbol list. As a bridge between text and speech, the speech token enables the text to be processed and understood more efficiently in the speech synthesis process, and converted into corresponding synthesized speech output. It is worth noting that due to the difference in the bit rate of the text and the bit rate of the speech signal, in actual applications, one text unit may correspond to multiple first speech units. Therefore, the speech conversion process based on the first model is performed on data of non-equivalent resolution, which is not necessary for models in the NLP field, especially large language models in the NLP field.

[0032] In the embodiment of the present application, both the first model and the second model can be large language models, such as large language models based on AR architecture. The difference between the two is that the first model is a lightweight large language model, and the second model is a large language model of larger magnitude, that is, the parameters of the first model are less than the parameters of the second model. Based on this, the duration of the first model performing speech conversion on the first text based on the first sequence is much shorter than the speech conversion duration of the second model, which is conducive to improving the speech synthesis efficiency of the first text.

[0033] In the embodiment of the present application, the above S102 can be implemented in various appropriate ways.

[0034] In one implementation, the first sequence is obtained by concatenating the text identifier of the first text and the first start identifier, and the first start identifier refers to the start identifier of the speech unit mentioned above. The above S102 includes the following steps: multiple sequences of the first sequence are performed by the first model to obtain multiple first speech units, and a first speech unit sequence is generated based on the multiple first speech units. In the first prediction process, the first model predicts the first first speech unit after the first start identifier based on the first sequence; in the kth prediction process, the first model predicts the kth first speech unit based on the first sequence and the first k-1 first speech units, where k is an integer greater than 1.

[0035] For example, assuming that the first sequence is [spk][text begin][ttoken1, ttoken2, … , ttokenN][text end][mel begin], the first model predicts the first sequence 5 times. Among them, [spk] represents the acoustic feature of the speaker, [text begin] represents the start marker of the text unit, [ttoken1, ttoken2, … , ttokenN] represents the text unit of the first text, [text end] represents the end marker of the text unit, and [mel begin] represents the first start marker.

[0036] In the first prediction process, the first sequence is input into the first model to obtain the first first speech unit mtoken1 after [mel begin].

[0037] In the second prediction process, the first sequence and mtoken1 are concatenated and input into the first model to obtain the second first speech unit mtoken2.

[0038] By analogy, the third prediction obtains the third first speech unit mtoken3, the fourth prediction obtains the fourth first speech unit mtoken4, and the fifth prediction obtains the fifth first speech unit mtoken5. These first speech units are concatenated to obtain the first speech unit sequence [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5].

[0039] The above implementation is applicable to the case where the structure of the first model is similar to that of the second model, for example, the first model is obtained by simplifying the structure of the second model.

[0040] In another implementation, the first sequence is obtained by concatenating the text identifier of the first text and the first start identifier. Before the above S102, the method provided in the embodiment of the present application also includes: predicting the first second speech unit after the first start identifier based on the first sequence through the second model. In this case, the above S102 includes the following steps: predicting the first sequence multiple times through the first model to obtain multiple first speech units, and generating a first speech unit sequence based on multiple first speech units. Among them, in the first prediction process, the first model predicts the first first speech unit after the first second speech unit based on the first sequence and the first second speech unit; in the kth prediction process, the first model predicts the kth first speech unit based on the first sequence, the first second speech unit and the first k-1 first speech units, where k is an integer greater than 1.

[0041] For example, still taking the above-mentioned first sequence as an example, it is assumed that the first model makes 5 predictions for the first sequence.

[0042] First, the first sequence is input into the second model to obtain the first second speech unit mtoken1' after [mel begin]. Then, multiple predictions are made based on the first sequence and mtoken1' through the first language model. The specific process is as follows: In the first prediction process, the first sequence and mtoken1' are input into the first model to obtain the first first speech unit mtoken1 after mtoken1'.

[0043] In the second prediction process, the first sequence, mtoken1' and mtoken1 are concatenated and input into the first model to obtain the second first speech unit mtoken2.

[0044] By analogy, the third prediction obtains the third first speech unit mtoken3, the fourth prediction obtains the fourth first speech unit mtoken4, and the fifth prediction obtains the fifth first speech unit mtoken5. These first speech units are concatenated to obtain the first speech unit sequence [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5].

[0045] Due to the high sensitivity to speech units in the speech synthesis process, if the quality of the speech unit is poor, the final synthesized speech will perform poorly, affecting user perception. In addition, the model predicts the next speech unit based on the existing speech unit. If the quality of the first speech unit is poor, the quality of all subsequent speech units will be affected. To this end, in the above implementation method, the first second speech unit is first predicted by the second model with high prediction accuracy, and then the first model predicts multiple subsequent first speech units based on the first second speech unit. This can ensure the accuracy of the first speech unit sequence and provide reliable data support for the subsequent generation of high-quality synthesized speech.

[0046] The above implementation is applicable to the case where the first model and the second model have significant structural differences.

[0047] The above shows some implementations of S102. Of course, it should be understood that the above S102 can also be implemented in other ways, which are not limited in the embodiments of the present application.

[0048] In the embodiment of the present application, the number of times the first input unit is predicted by the first model can be set according to actual needs, and the embodiment of the present application does not limit this. The number can also be called the size of the speculation window.

[0049] In one implementation, the number of times the first input unit is predicted by the first model is not limited until the first speech unit sequence contains a target first speech unit indicating a first end marker, wherein the first end marker is used to clearly indicate the end of the first speech unit.

[0050] In another implementation, in order to better balance the accuracy and efficiency of the first model in voice conversion, the number of times the first model is used to predict the first input unit can be determined based on the structural similarity between the first model and the second model. For example, if the first model and the second model are similar in structure, the above number of times can be set to a larger value, such as 5 times; if the first model and the second model are very different in structure, the above number of times can be set to a smaller value, such as 2 times.

[0051] In another implementation, the number of times can also be determined according to the training effect of the first model. For example, if the first model has more training data, is fully trained, and has a better training effect, the number of times can be set to a larger value; otherwise, the number of times can be set to a smaller value.

[0052] S104, performing speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence.

[0053] Normally, the second model directly performs speech conversion on the first text based on the first speech unit sequence, which requires multiple iterative predictions. Since the second model is a larger model, its parameters are more than those of the first model, so the conversion time required is longer. In the above S104, since the model predicts the next speech unit using the obtained speech unit and uses matrix calculation, based on the first speech unit sequence obtained by the lightweight model, the second model can achieve multiple prediction effects with the first model after one prediction, and obtain the second speech unit sequence.

[0054] In one implementation, the above S104 includes the following steps: concatenating the first sequence and the first speech unit sequence to obtain a second sequence; predicting the second sequence through a second model to obtain a first output sequence, the first output sequence including: the next fourth speech unit of the first start identifier, the next fourth speech unit of each first speech unit in the first speech unit sequence; sampling the first output sequence to obtain multiple second speech units after the first start identifier; generating a second speech unit sequence based on the multiple second speech units.

[0055] For example, assuming that the first sequence is [spk][text begin][ttoken1,ttoken2,… ,ttokenN][text end][mel begin], and the first speech unit sequence is [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5], concatenate the two to obtain the second sequence: [spk][text begin][ttoken1,ttoken2,… ,ttokenN][text end][mel begin] [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5].

[0056] The second sequence is input into the second model, and the second model predicts the probability distribution of the next unit of each information in the second sequence, and the following first output sequence is obtained: [NXT spk][NXT text begin][NXT ttoken1] …[NXT mel begin][NXT mtoken1][NXT mtoken2],…,[NXT mtoken5]. Among them, [NXT spk] represents the next unit predicted for [spk]; [NXTtext begin] represents the next unit predicted for [text begin]; [NXT ttoken1] represents the next unit predicted for [ttoken1]; [NXT mel begin] represents the next unit predicted for [mel begin], also known as the next fourth speech unit of the first start marker; [NXT mtoken1] represents the next unit predicted for [mtoken1], also known as the next fourth speech unit of [mtoken1]; [NXT mtoken2] represents the next unit predicted for [mtoken2], also known as the next fourth speech unit of [mtoken2], and [NXT mtoken5] represents the next unit predicted for [mtoken5], also known as the next speech unit of [mtoken5].

[0057] Further, [NXT mel begin] is sampled to obtain the first second speech unit [mtoken1'] after [NXT mel begin]; [NXT mtoken1] is sampled to obtain the second second speech unit [mtoken2'] after [NXT mel begin]; and so on, 5 second speech units are obtained, and based on these second speech units, a second speech unit sequence [mtoken1'][mtoken2'][mtoken3'][mtoken4'][mtoken5'] is generated.

[0058] It can be seen that the second model can achieve multiple prediction effects of the first model after one prediction, and obtain the second speech unit sequence.

[0059] In another implementation, if the second model is used to first predict the first second speech unit, and then the first model is used to predict multiple first speech units after the second speech unit, the above S104 includes the following steps: concatenating the first sequence, the first second speech unit and the first speech unit sequence to obtain a second sequence; outputting the second sequence through the second model to obtain a first output sequence, the first output sequence including: the next fourth speech unit of the first start marker, the next fourth speech unit of each first speech unit in the first speech unit sequence; sampling the first output sequence to obtain multiple second speech units after the first start marker; generating a second speech unit sequence based on multiple second speech units.

[0060] Since the specific process of the above implementation is similar to that of the first implementation, it will not be described in detail. It is worth noting that in this implementation, when sampling multiple second speech units, the first second speech unit that has been generated is not included in the sampling.

[0061] The embodiment of the present application shows some implementations of the above S104. Of course, it should be understood that the above S104 can also be implemented in other ways, which are not limited in the embodiment of the present application.

[0062] S106: Generate a third speech unit sequence based on the matching result between the first speech unit sequence and the second speech unit sequence.

[0063] Matching the first speech unit sequence with the second speech unit sequence is to verify the first speech unit. The third speech unit sequence includes at least one third speech unit, and the third speech unit refers to a verified speech unit that can be used to synthesize the speech of the first text. Specifically, for each first speech unit in the first speech unit sequence, the first speech unit can be matched with the second speech unit at the same position in the second speech unit sequence. If the two match, for example, the similarity between the two is greater than the similarity threshold, it is determined that the first speech unit has passed the verification and is used as the third speech unit.

[0064] In the embodiment of the present application, the above S106 can be implemented in various appropriate ways.

[0065] In one implementation, based on the matching result between the first speech unit sequence and the second speech unit sequence, it is determined whether there is a target first speech unit in the first speech unit sequence, wherein the target first speech unit is the first first speech unit that does not match the second speech unit at the same position in the second speech unit sequence; if there is no target first speech unit in the first speech unit sequence, the first speech unit sequence and the next fourth speech unit of the last speech unit in the first speech unit sequence are concatenated as the third speech unit to obtain a third speech unit sequence; if there is a target first speech unit in the first speech unit sequence, the first speech unit before the target first speech unit and the target second speech unit are concatenated as the third speech unit to obtain a third speech unit sequence, wherein the target second speech unit is the second speech unit in the second speech unit sequence with the same position as the target first speech unit.

[0066] For example, assuming that the first speech unit sequence is [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5], the second speech unit sequence is [mtoken1'][mtoken2'][mtoken3'][mtoken4'][mtoken5'], the next fourth speech unit after [mtoken5'] is [NXT mtoken5].

[0067] In the optimal case, each first speech unit in the first speech unit sequence matches the second speech unit at the same position in the second speech unit, and then each first speech unit and [NXT mtoken5] are used as the third speech unit to obtain the third speech unit sequence: [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5][NXTmtoken5]. In this case, the second model only makes one prediction to obtain 6 valid third speech units. Further, in the next speculative sampling process, the first model predicts the 7th valid third speech unit and the third speech units thereafter based on the third speech unit sequence.

[0068] In the suboptimal case, the third first speech unit [mtoken3] in the first speech unit sequence is the first first speech unit that does not match the third second speech unit [mtoken3'] in the second speech unit sequence, and then the first two first speech units and the third second speech unit are concatenated as the third speech unit to obtain the third speech unit sequence: [mtoken1][mtoken2][mtoken3']. In this case, the second model obtains three valid third speech units with only one prediction. Furthermore, in the next speculative sampling process, the first model predicts the fourth valid third speech unit and the third speech units thereafter based on the third speech unit sequence.

[0069] In the worst case, the first speech unit sequence does not match the second speech unit sequence, and the first second speech unit is used as the third speech unit, resulting in the third speech unit sequence: [mtoken1']. In this case, the second model obtains a valid third speech unit after one prediction. Furthermore, in the next speculative sampling process, the first model predicts the second valid third speech unit and the third speech unit thereafter based on the third speech unit sequence.

[0070] In another implementation, based on the matching result between the first speech unit sequence and the second speech unit sequence, it is determined whether the target first speech unit exists in the first speech unit sequence; if the target first speech unit does not exist in the first speech unit sequence, the first speech unit sequence is used as the third speech unit sequence; if the target first speech unit exists in the first speech unit sequence, the first speech unit before the target first speech unit and the target second speech unit are concatenated as the third speech unit to obtain a third speech unit sequence.

[0071] The specific process of this implementation is similar to that of the first implementation and will not be repeated here.

[0072] The embodiment of the present application shows some implementations of the above S106. Of course, it should be understood that the above S106 can also be implemented in other ways, which are not limited in the embodiment of the present application.

[0073] S108: Generate synthesized speech of the first text based on the third speech unit sequence.

[0074] Since the model stops speech conversion based on the first end marker (i.e., the end marker of the speech unit sequence), in order to accurately control the length and quality of the third speech unit sequence used to synthesize the first text, the above S108 includes the following steps: determining whether the third speech unit sequence contains the target third speech unit representing the first end marker; if the third speech unit sequence contains the target third speech unit, concatenating the first sequence with the third speech unit sequence to obtain a new first sequence, and repeatedly executing S102 to S108 until the third speech unit sequence contains the target third speech unit; if the third speech unit sequence contains the target third speech unit, generating the synthesized speech of the first text based on the third speech unit sequence.

[0075] For example, assuming that the number of predictions of the first model is 5 times, after the first round of speech conversion, the third speech unit sequence [mtoken1][mtoken2][mtoken3][mtoken4][mtoken5][NXT mtoken5] is obtained. Further, the third speech unit sequence is concatenated after the first sequence to obtain a new first sequence, and the above S102 to S108 are repeatedly executed until the obtained third speech unit sequence contains the target third speech unit [mel end]. Further, post-processing is performed based on all the third speech unit sequences to convert them into the synthesized speech of the first text. At this point, the speech synthesis of the first text is completed.

[0076] The speech synthesis method provided in the embodiment of the present application adopts the technical idea of ​​speculative sampling and adopts a speech synthesis scheme that uses both the main and auxiliary models. Specifically, a lightweight model with fewer parameters is first used to perform speech conversion on the first text based on a first sequence of text units containing the first text to obtain a first speech unit sequence, and then a heavyweight model with more parameters is used to perform a speech conversion on the first text with the help of the first sequence and the first speech unit sequence to obtain a second speech unit sequence, and the first speech unit sequence is used to verify the first speech unit sequence, and reliable speech units are determined and adopted; further, starting from the speech units that have not been adopted, the above operation is repeated until all speech units corresponding to the first text are obtained, and the synthesized speech of the text is obtained based on these speech units.

[0077] Since the parameters of the lightweight model are much smaller than those of the heavyweight model, the prediction time of the lightweight model is much shorter than that of the heavyweight model; secondly, since the model uses the obtained speech units to predict the next speech unit and uses matrix calculation, based on the speech unit sequence output by the existing lightweight model, the heavyweight model can complete the verification of the speech unit sequence output by the lightweight model after one prediction, which significantly reduces the time consumption compared to the traditional multi-step prediction scheme. Moreover, although the reliability of the lightweight model is not as good as that of the heavyweight model, the quality of the verified speech unit can be ensured after verification by the heavyweight model. It can be seen that the speech synthesis method proposed in the embodiment of the present application significantly improves the speech synthesis efficiency while ensuring the accuracy of speech synthesis, and can meet the needs of real-time speech synthesis.

[0078] The present application also provides a method for training a first model. Figure 2 , is a flow chart of a first model training method provided in an embodiment of the present application, the method comprising the following steps: S202, obtaining a third model, and obtaining a first sample sequence used for training the second model and a first sample speech unit sequence corresponding to the first sample sequence.

[0079] The first sample sequence includes text units of the second text. The first sample speech unit sequence includes a plurality of first sample speech units used to obtain the synthesized speech of the second text. The first sample sequence and its corresponding first sample speech unit sequence are training data for the second model. Exemplarily, Figure 3A schematic diagram of training data of a second model is shown, in which speaker represents the acoustic features of the speaker, text begin represents the start mark of the text unit of the second text, text end represents the end mark of the text unit of the second text, and the text units of the second text are between the two, mel begin represents the second start mark, which is used to indicate the position of the first first sample speech unit, and mel end represents the second end mark, which is used to indicate the position of the last sample speech unit, and the text units of the second text are between the two. The sequence composed of text begin, the text unit of the second text, text end and mel begin is the first sample sequence, and the sequence composed of multiple first sample speech units and mel end is the first sample speech unit sequence.

[0080] The third model refers to a lightweight large language model to be trained. The first model can have various appropriate structures, which are not limited in the embodiments of the present application.

[0081] In one embodiment, the third model has a similar structure to the second model. Figure 4 As shown, the second model includes n second conversion layers, where n is an integer greater than 1. The second conversion layer may use a transformer. Optionally, the second model may also include an embedding layer and a prediction layer. The embedding layer is used to convert the sequence of the second model into a corresponding vector representation, and the n second conversion layers are used to determine the relationship between each information in the sequence based on the vector representation and the attention mechanism to obtain a latent vector sequence; the prediction layer is used to obtain a corresponding speech unit sequence based on the latent vector sequence.

[0082] The third model can be obtained by cutting the structure of the second model. Specifically, Figure 5 As shown, the first model includes m first conversion layers, m is an integer greater than 1, and m is less than n. The value of m can be set according to actual needs, for example, m=3. Based on the parameters of the 1st second conversion layer, the parameters of the 1st first conversion layer are initialized; based on the parameters of the nth second conversion layer, the parameters of the mth first conversion layer are initialized; based on the parameters of the jth second conversion layer, the parameters of the ith first conversion layer are initialized, wherein i and j are both integers greater than 1, and i<m, j<n. Exemplarily, the parameters of the 6th second conversion layer are used to initialize the parameters of the 3rd first conversion layer. In addition, the parameters of the embedding layer of the second model can also be used to initialize the parameters of the embedding layer of the first model.

[0083] In practical applications, each first conversion layer may be initialized by random initialization using a normal distribution.

[0084] Since the structure of the third model is similar to that of the second model, the parameters of the third model are initialized using the parameters of the second model, which can improve the prediction accuracy of the third model. In addition, since the first conversion layer is close to the text, it has a stronger ability to learn the contextual relationship between text units. Using the parameters of the first second conversion layer to initialize the parameters of the first first conversion layer can ensure that the third model also has the ability to accurately learn the contextual relationship between text units; the last conversion layer mainly learns the association between text units and speech units, which greatly affects the prediction accuracy of speech units. Using the parameters of the last second conversion layer to initialize the parameters of the last first conversion layer can ensure that the third model also has the ability to accurately learn the association between text units and speech units; the conversion layer in the middle plays a transition role. Using the parameters of the middle second conversion layer to initialize the parameters of the middle first conversion layer can improve the transition ability between each conversion layer in the third model.

[0085] In another embodiment, the third model has a different structure from the second model. Figure 6 As shown, the third model adopts a simplified sequence prediction model, which includes a convolutional layer, a first long short-term memory network (Long Short-Term Memory, LSTM), a first normalization and pooling layer, a second long short-term memory network, a second normalization and pooling layer, and a classification layer.

[0086] The convolution layer is used for channel compression. Since the inner layer hidden variable dimension of the third model is high, it is not conducive to sequence prediction learning. The convolution layer can compress the dimension of the hidden variable.

[0087] The first LSTM, the first normalization and pooling layer, the second LSTM, and the second normalization and pooling layer are used for context learning. The first LSTM and the second LSTM are mainly used to learn the long-term and short-term dependencies of the sequence. The output of the first LSTM is regularized and pooled by the first normalization and pooling layer as the input of the second LSTM. The output of the second LSTM is regularized and pooled by the second normalization and pooling layer as the input of the classification layer. The classification layer includes a softmax activation layer and a linear layer, which are used to predict the speech unit based on the output of the second LSTM to obtain the corresponding speech unit prediction.

[0088] S204, performing speech conversion on the second text based on the first sample sequence through a third model to obtain a second sample speech unit sequence.

[0089] The second sample speech unit sequence includes a plurality of second sample speech units arranged in sequence.

[0090] In one implementation, if the structure of the third model is similar to that of the second model, such as Figure 5 The third model shown in the figure predicts the first sample sequence multiple times through the third model until the obtained second sample speech unit sequence contains the second sample speech unit indicating the second end mark. In the first prediction process, the third model predicts the first second sample speech unit based on the first sample sequence; in the kth prediction process, the third model predicts the kth second sample speech unit based on the first sample sequence and the first k-1 second sample speech units, where k is an integer greater than 1.

[0091] In another implementation, if the structure of the third model is significantly different from that of the second model, such as Figure 6 The third model shown, the first sample speech unit sequence also includes a second end marker after the last first sample speech unit. In this case, the above S204 includes: splicing the first sample sequence and the first sample speech unit sequence and inputting them into the third model for prediction, and obtaining the second output sequence of the nth second conversion layer, the second output sequence includes: a second start marker, and multiple latent vectors after the second start marker; splicing the second start marker and the multiple latent vectors to obtain a second sample sequence; using the third model, performing speech conversion on the second text based on the second sample sequence to obtain a second sample speech unit sequence.

[0092] For example, Figure 7 As shown, the first sample sequence includes: spk, text begin, a text unit of the second text, text end and mel begin (second start marker), and the first sample speech unit sequence includes: multiple first sample speech units and mel end. After the first sample sequence and the first sample speech unit sequence are concatenated, a sequence with a total length of L is obtained, and the sequence is input into the second model for prediction, and the second output sequence of the nth second conversion layer is obtained. The total length of the second output sequence is L, and a subsequence from mel begin to mel end is intercepted therefrom to obtain a second sample sequence with a length of subL. Further, the second sample sequence is input into the third model for multiple predictions to obtain a second sample speech unit sequence.

[0093] Since the second model has a high prediction accuracy, the second output sequence predicted from the last second conversion layer of the second model for the first sample sequence and the first sample speech unit sequence has a high accuracy, which implies the association between the text unit of the second text and the first sample speech unit, and plays an auxiliary role in the training of the third model, which is conducive to improving the training effect of the third model. In addition, adding a second end mark in the first sample speech unit sequence and adding a second start mark in the second sample sequence can realize the alignment of the first sample speech unit sequence and the second sample sequence, so that the third model can better learn the association between the text unit and the speech unit, and further improve the training effect of the third model.

[0094] S206: Based on the matching result between the first sample speech unit sequence and the second sample speech unit sequence, adjust the parameters of the third model to obtain the first model.

[0095] Specifically, based on the difference between the first sample speech unit sequence and the second sample speech unit sequence, the loss of the third model is determined; then, the parameters of the third model are adjusted using a back propagation algorithm, a gradient descent algorithm, etc.

[0096] The above S202~S206 is only a training process for the third model. In practical applications, the third model can be trained multiple times until the training stop condition is met. Among them, the training stop condition can be set according to actual needs, for example, the number of training times reaches a threshold, or the third model converges, or the loss of the third model is less than a loss threshold, etc., which is not limited in the embodiment of the present application.

[0097] The training method for the first model provided in the embodiment of the present application abandons the traditional distillation technology or bypass training technology, and instead uses the training data of the second model to train the third model to obtain the first model. This can reduce the loss generated by the first model in the prediction process and improve the prediction accuracy of the first model, thereby improving the accuracy of speech synthesis based on the first model.

[0098] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0099] Based on the same inventive concept, the present application also provides a speech synthesis device. Figure 8, is a structural diagram of a speech synthesis device 800 provided in an embodiment of the present application, the device 800 includes: a first generation module 810, a second generation module 820, a third generation module 830 and a speech synthesis module 840.

[0100] The first generating module 810 is used to perform speech conversion on the first text based on the first sequence through a first model to obtain a first speech unit sequence, where the first sequence includes text units of the first text.

[0101] The second generation module 820 is used to perform speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence; the parameters of the first model are less than the parameters of the second model.

[0102] The third generating module 830 is used to generate a third speech unit sequence based on the matching result between the first speech unit sequence and the second speech unit sequence.

[0103] The speech synthesis module 840 is used to generate synthesized speech of the first text based on the third speech unit sequence.

[0104] In another embodiment, the first sequence is obtained by concatenating the text unit of the first text and the first start marker; The second generation module is further configured to predict, by using the second model, a first second speech unit after the first start identifier based on the first sequence; The first generating module is used for: Predicting the first sequence multiple times by using the first model to obtain multiple first speech units, and generating the first speech unit sequence based on the multiple first speech units; Wherein, in the first prediction process, the first model predicts the first first speech unit after the first second speech unit based on the first sequence and the first second speech unit; In the kth prediction process, the first model predicts the kth first speech unit based on the first sequence, the first second speech unit and the first k-1 first speech units, where k is an integer greater than 1.

[0105] In another embodiment, the first sequence further includes a first start identifier; The second generating module is used for: concatenating the first sequence and the first speech unit sequence to obtain a second sequence; Predicting the second sequence by using the second model to obtain a first output sequence, wherein the first output sequence includes: the fourth speech unit next to the first start identifier, and the fourth speech unit next to each first speech unit in the first speech unit sequence; Sampling the first output sequence to obtain a plurality of second speech units after the first start identifier; Based on the plurality of second speech units, a second speech unit sequence is generated.

[0106] In another embodiment, the third generation module is used to: Based on the matching result, determining whether there is a target first speech unit in the first speech unit sequence, the target first speech unit being the first first speech unit that does not match the second speech unit at the same position in the second speech unit sequence; If the target first speech unit does not exist in the first speech unit sequence, the first speech unit sequence and the fourth speech unit next to the last speech unit in the first speech unit sequence are concatenated as the third speech unit to obtain a third speech unit sequence; If the target first speech unit exists in the first speech unit sequence, the first speech unit before the target first speech unit and the target second speech unit are concatenated as the third speech unit to obtain a third speech unit sequence, wherein the target second speech unit is the second speech unit in the second speech unit sequence with the same position as the target first speech unit.

[0107] In another embodiment, the speech synthesis module is used to: Determining whether the third speech unit sequence includes a target third speech unit indicating a first end marker; If the third speech unit sequence includes the target third speech unit, the first sequence is concatenated with the third speech unit sequence to obtain a new first sequence, and the first generation module, the second generation module and the third generation module are called to repeatedly perform the steps of performing speech conversion on the first text based on the first sequence through the first model to determining whether the third speech unit sequence includes the target third speech unit indicating the first end mark, until the third speech unit sequence includes the target third speech unit; If the third speech unit sequence includes the target third speech unit, then generating synthesized speech of the first text based on the third speech unit sequence.

[0108] In another embodiment, the first model is obtained by: Acquire a third model, and acquire a first sample sequence used to train the second model and a first sample speech unit sequence corresponding to the first sample sequence, wherein the first sample sequence includes text units of a second text, and the first sample speech unit sequence includes a plurality of first sample speech units used to obtain synthesized speech of the second text; Performing speech conversion on the second text based on the first sample sequence by the third model to obtain a second sample speech unit sequence, wherein the second sample speech unit sequence includes a plurality of second sample speech units arranged in sequence; Based on the matching result between the first sample speech unit sequence and the second sample speech unit sequence, the parameters of the third model are adjusted to obtain the first model.

[0109] In another embodiment, the third model includes m first conversion layers, the second model includes n second conversion layers, m is an integer greater than 1, and m is less than n; Before performing speech conversion on the second text based on the first sample sequence through the third model to obtain a second sample speech unit sequence, the method further includes: Initializing parameters of the first first conversion layer based on parameters of the first second conversion layer; Initializing parameters of the mth first transformation layer based on parameters of the nth second transformation layer; Initialize the parameters of the i-th first conversion layer based on the parameters of the j-th second conversion layer, wherein i and j are both integers greater than 1, i<m, j<n.

[0110] In another embodiment, the second model includes n second conversion layers, where n is an integer greater than 1; the first sample speech unit sequence also includes a second end marker after the last first sample speech unit; The method of performing speech conversion on the second text based on the first sample sequence by the third model to obtain a second sample speech unit sequence includes: splicing the first sample sequence and the first sample speech unit sequence and inputting them into the second model for prediction, and Obtain a second output sequence of the nth second transformation layer, the second output sequence comprising: a second start identifier, and a plurality of latent vectors after the second start identifier; Concatenating the second start identifier and the multiple latent vectors to obtain a second sample sequence; The second text is converted into speech based on the second sample sequence by using the third model to obtain a second sample speech unit sequence.

[0111] Fig. 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Fig. 9 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0112] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0114] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a speech synthesis device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: Performing speech conversion on the first text based on the first sequence by using a first model to obtain a first speech unit sequence, wherein the first sequence includes text units of the first text; performing speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence; the parameters of the first model are less than the parameters of the second model; generating a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence; Based on the third speech unit sequence, synthesized speech of the first text is generated.

[0115] The above application Figure 1The method performed by the speech synthesis device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0116] The electronic device may also perform Figure 1 The method and the speech synthesis device are implemented in Figure 1 , Figure 2 The functions of the embodiments shown in the figure will not be described in detail in the embodiments of the present application.

[0117] Of course, in addition to software implementation, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0118] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations: Performing speech conversion on the first text based on the first sequence by using a first model to obtain a first speech unit sequence, wherein the first sequence includes text units of the first text; performing speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence; the parameters of the first model are less than the parameters of the second model; generating a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence; Based on the third speech unit sequence, synthesized speech of the first text is generated.

[0119] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps in the speech synthesis method provided in the embodiment of the present application.

[0120] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0121] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0122] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0123] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0124] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A speech synthesis method, characterized in that: include: Performing speech conversion on the first text based on the first sequence by using a first model to obtain a first speech unit sequence, wherein the first sequence includes text units of the first text; performing speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence; the parameters of the first model are less than the parameters of the second model; generating a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence; Based on the third speech unit sequence, synthesized speech of the first text is generated.

2. The method according to claim 1, characterized in that The first sequence is obtained by concatenating the text unit of the first text and the first start marker; Before performing speech conversion on the first text based on the first sequence by using the first model to obtain the first speech unit sequence, the method further includes: Predicting, by the second model, a first second speech unit after the first start marker based on the first sequence; The method of performing speech conversion on the first text based on the first sequence by using the first model to obtain a first speech unit sequence includes: Predicting the first sequence multiple times by using the first model to obtain multiple first speech units, and generating the first speech unit sequence based on the multiple first speech units; Wherein, in the first prediction process, the first model predicts the first first speech unit after the first second speech unit based on the first sequence and the first second speech unit; In the kth prediction process, the first model predicts the kth first speech unit based on the first sequence, the first second speech unit and the first k-1 first speech units, where k is an integer greater than 1.

3. The method according to claim 1, characterized in that The first sequence also includes a first start identifier; The method of performing speech conversion on the first text based on the first speech unit sequence and the first sequence by using the second model to obtain a second speech unit sequence includes: concatenating the first sequence and the first speech unit sequence to obtain a second sequence; Predicting the second sequence by using the second model to obtain a first output sequence, wherein the first output sequence includes: the fourth speech unit next to the first start identifier, and the fourth speech unit next to each first speech unit in the first speech unit sequence; Sampling the first output sequence to obtain a plurality of second speech units after the first start identifier; Based on the plurality of second speech units, a second speech unit sequence is generated.

4. The method according to claim 3, characterized in that The generating a third speech unit sequence based on the matching result between the first speech unit sequence and the second speech unit sequence comprises: Based on the matching result, determining whether there is a target first speech unit in the first speech unit sequence, the target first speech unit being the first first speech unit that does not match the second speech unit at the same position in the second speech unit sequence; If the target first speech unit does not exist in the first speech unit sequence, the first speech unit sequence and the fourth speech unit next to the last speech unit in the first speech unit sequence are concatenated as the third speech unit to obtain a third speech unit sequence; If the target first speech unit exists in the first speech unit sequence, the first speech unit before the target first speech unit and the target second speech unit are concatenated as the third speech unit to obtain a third speech unit sequence, wherein the target second speech unit is the second speech unit in the second speech unit sequence with the same position as the target first speech unit.

5. The method according to claim 1, characterized in that: The step of generating a synthesized speech of the first text based on the third speech unit sequence comprises: Determining whether the third speech unit sequence includes a target third speech unit indicating a first end marker; If the third speech unit sequence includes the target third speech unit, the first sequence is concatenated with the third speech unit sequence to obtain a new first sequence, and the steps of performing speech conversion on the first text based on the first sequence through the first model to determining whether the third speech unit sequence includes the target third speech unit indicating the first end mark are repeatedly performed until the third speech unit sequence includes the target third speech unit; If the third speech unit sequence includes the target third speech unit, then generating synthesized speech of the first text based on the third speech unit sequence.

6. The method according to claim 1, characterized in that The first model is obtained by: Acquire a third model, and acquire a first sample sequence used to train the second model and a first sample speech unit sequence corresponding to the first sample sequence, wherein the first sample sequence includes text units of a second text, and the first sample speech unit sequence includes a plurality of first sample speech units used to obtain synthesized speech of the second text; Performing speech conversion on the second text based on the first sample sequence by the third model to obtain a second sample speech unit sequence, wherein the second sample speech unit sequence includes a plurality of second sample speech units arranged in sequence; Based on the matching result between the first sample speech unit sequence and the second sample speech unit sequence, the parameters of the third model are adjusted to obtain the first model.

7. The method according to claim 6, characterized in that The third model includes m first conversion layers, the second model includes n second conversion layers, m is an integer greater than 1, and m is less than n; Before performing speech conversion on the second text based on the first sample sequence through the third model to obtain a second sample speech unit sequence, the method further includes: Initializing parameters of the first first conversion layer based on parameters of the first second conversion layer; Initializing parameters of the mth first transformation layer based on parameters of the nth second transformation layer; Initialize the parameters of the i-th first conversion layer based on the parameters of the j-th second conversion layer, wherein i and j are both integers greater than 1, i<m, j<n.

8. The method according to claim 6, characterized in that The second model includes n second conversion layers, where n is an integer greater than 1; the first sample speech unit sequence also includes a second end marker after the last first sample speech unit; The method of performing speech conversion on the second text based on the first sample sequence by the third model to obtain a second sample speech unit sequence includes: splicing the first sample sequence and the first sample speech unit sequence and inputting them into the second model for prediction, and Obtain a second output sequence of the nth second transformation layer, the second output sequence comprising: a second start identifier, and a plurality of latent vectors after the second start identifier; Concatenating the second start identifier and the multiple latent vectors to obtain a second sample sequence; The second text is converted into speech based on the second sample sequence by using the third model to obtain a second sample speech unit sequence.

9. A speech synthesis device, characterized in that: include: A first generating module, configured to perform speech conversion on a first text based on a first sequence by using a first model to obtain a first speech unit sequence, wherein the first sequence includes text units of the first text; A second generating module is used to perform speech conversion on the first text based on the first speech unit sequence and the first sequence through a second model to obtain a second speech unit sequence; the parameters of the first model are less than the parameters of the second model; A third generating module, configured to generate a third speech unit sequence based on a matching result between the first speech unit sequence and the second speech unit sequence; A speech synthesis module is used to generate synthesized speech of the first text based on the third speech unit sequence.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the speech synthesis method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the speech synthesis method as claimed in any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the speech synthesis method according to any one of claims 1 to 8.

Images