Speech unit prediction model training method, speech synthesis method and electronic equipment

Abstract

One or more embodiments of the invention provide a training method of a voice unit prediction model, a voice synthesis method and electronic equipment. The training method comprises the following steps: acquiring a first training sample and a second training sample; wherein the first training sample comprises a plurality of texts, and the second training sample comprises a plurality of texts and voice units thereof; the number of texts in the first training sample is greater than the number of texts in the second training sample; performing self-supervised pre-training on a preset neural network model by using the first training sample to obtain a language model for learning language expression information from the first training sample; performing supervised training on the language model by using the second training sample to obtain a voice unit prediction model; the voice unit prediction model is used for processing an input text to obtain a voice unit sequence. According to the embodiment of the invention, pre-training is performed to obtain the language model learning the language expression information, and then supervised training is performed on the language model, so that the prediction accuracy of the voice unit prediction model is improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of terminal technology, and in particular, to a training method for a speech unit prediction model, a speech synthesis method, and an electronic device. Background Art

[0002] With the rapid development of voice interaction technology in recent years, it has gradually entered people's daily life. Among them, speech synthesis technology, as an important part of voice interaction technology, plays an important role in the voice interaction process.

[0003] Text to Speech (TTS) is a technology that converts text information generated by a computer or input externally into understandable and fluent spoken output. The speech synthesis process is to first convert the text into a sequence of speech units, and then generate a speech waveform based on the sequence of speech units. The first step of converting text into a sequence of speech units involves linguistic processing, which generally requires a whole chain of word segmentation, word-to-phonetic conversion, text regularization, rhythm prediction, and polyphone prediction; the second step requires speech synthesis technology to synthesize a high-quality speech stream. Among them, the process of converting text into a sequence of speech units during speech synthesis is relatively complicated and involves multiple links. If the processing link is too long, it will lead to error accumulation problems, making the synthesis prone to errors and reducing the sound quality. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a training method for a speech unit prediction model, a speech synthesis method, a server, a client, an electronic device, and a computer-readable storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for training a speech unit prediction model is proposed, comprising:

[0007] Obtain a first training sample and a second training sample; wherein the first training sample includes a plurality of texts, and the second training sample includes a plurality of texts and their speech units; the number of texts in the first training sample is greater than the number of texts in the second training sample;

[0008] Using the first training sample to perform self-supervised pre-training on a preset neural network model to obtain a language model that learns language expression information from the first training sample;

[0009] The language model is supervisedly trained using the second training sample to obtain a speech unit prediction model; wherein the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0010] Beneficial effect: In this embodiment, the neural network model can be pre-trained by self-supervised learning. The self-supervised learning process does not require manual annotation, and a large amount of text can be obtained for pre-training, so that the neural network model can fully learn the language expression information and obtain a language model; on this basis, it is only necessary to use an appropriate number of second training samples to conduct supervised training on the language model, so that the language model that has learned the language expression information can further learn the mapping relationship between text and speech units, and finally obtain a speech unit prediction model. When the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, which is also beneficial to reduce the annotation cost.

[0011] Optionally, the using the first training sample to perform self-supervised pre-training on a preset neural network model includes:

[0012] Replacing the text at random positions in the first training sample with mask symbols to obtain masked text;

[0013] Using the masked text as input and the pre-masked text as a label, the following process is executed in a loop until the loop end condition is met: using the current neural network model to predict the text at the mask position in the masked text to obtain the predicted text; if the loop end condition is not met, adjusting the parameters of the current neural network model according to the predicted text and the pre-masked text to obtain an adjusted neural network model; wherein the adjusted neural network model is used as the current neural network model of the next loop process;

[0014] When the loop end condition is met, the current neural network model in the last loop process is used as the language model.

[0015] According to a second aspect of one or more embodiments of this specification, a method for training a speech unit prediction model is proposed, comprising:

[0016] Obtain a first training sample, a second training sample and a first neural network model; wherein the first training sample includes a plurality of texts, the second training sample includes a plurality of texts and their speech units, and the number of texts in the first training sample is greater than the number of texts in the second training sample; the first neural network model learns language expression information through a self-supervised pre-training process;

[0017] Performing knowledge distillation training on a preset second neural network model based on the first training sample and the first neural network model to obtain a language model that learns language expression information from the first neural network model; the scale of the language model is smaller than the scale of the first neural network model;

[0018] The language model is supervisedly trained using the second training sample to obtain a speech unit prediction model; wherein the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0019] Beneficial effects: This embodiment takes into account that there may be a first neural network model that has learned language expression information in the related technology, but it may not be suitable for the application scenario of the speech unit prediction model due to reasons such as its large scale. Therefore, the language expression information can be transferred from the large-scale first neural network model to the small-scale language model through the knowledge distillation process to adapt to the application scenario of the speech unit prediction model; on this basis, it is only necessary to use an appropriate number of second training samples to conduct supervised training on the language model, so that the language model that has learned the language expression information can further learn the mapping relationship between text and speech units, and finally obtain the speech unit prediction model. When the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, which is also beneficial to reduce the annotation cost.

[0020] Optionally, the first training sample includes at least a portion of text used for training the first neural network model;

[0021] The step of performing knowledge distillation training on a preset second neural network model based on the first training sample pair and the first neural network model to obtain a language model that learns language expression information from the neural network model includes:

[0022] Replacing the text at random positions in the first training sample with mask symbols to obtain masked text;

[0023] The following steps are executed in a loop until the loop end condition is met: using the current second neural network model to predict the text at the mask position in the masked text to obtain the first predicted text; and using the first neural network model to predict the text at the mask position in the masked text to obtain the second predicted text; if the loop end condition is not met, adjusting the parameters of the current second neural network model according to the first predicted text, the second predicted text and the text before masking to obtain the adjusted second neural network model; wherein the adjusted second neural network model is used as the current second neural network model of the next loop process;

[0024] When the loop end condition is met, the current second neural network model in the last loop process is used as the language model.

[0025] Optionally, the speech unit prediction model is used to be deployed to a client or a server, and the scale of the language model is determined according to the computing resources of the client or the server, and the scale of the language model is positively correlated with the computing resources of the client or the server.

[0026] Optionally, the language model includes at least one of a Transformer model, a BERT model and a GPT model; and / or some texts in the first training sample include texts obtained from the Internet using crawler technology and texts obtained from the developer's internal business data; some texts in the second training sample include texts obtained from the developer's internal business data.

[0027] According to a third aspect of an embodiment of the present disclosure, there is provided a speech synthesis method, comprising:

[0028] Get the text to be processed;

[0029] The text to be processed is converted into a speech unit sequence using a speech unit prediction model; wherein the speech unit prediction model is trained based on the training method described in the first aspect or the second aspect;

[0030] Processing the speech unit sequence using an acoustic model to obtain acoustic features;

[0031] The acoustic features are converted into speech data using a vocoder.

[0032] Beneficial effect: This embodiment applies the speech unit prediction model obtained through the above training in the speech synthesis process. The speech unit prediction model can be used to convert text into a speech unit sequence at one time without going through multiple steps such as word segmentation, character-phonetic conversion, text regularization, rhythm prediction and polyphonetic character prediction. This simplifies the process of converting text into a speech unit sequence, which is beneficial to reducing error accumulation and improving the success rate of speech synthesis. At the same time, the one-time conversion process is also beneficial to improving the efficiency of speech synthesis.

[0033] Optionally, the acoustic model includes a decoder, a parameter adjuster and an encoder;

[0034] The decoder is used to process the speech unit sequence to obtain a representation vector representing contextual semantic information;

[0035] The parameter adjuster is used to process the representation vector to obtain speech information, wherein the speech information includes at least one of the duration, volume and pitch of the speech;

[0036] The decoder is used to process the representation vector and the speech information to obtain acoustic features.

[0037] Optionally, the encoder is formed by stacking multiple layers of Transformer components; the parameter adjuster includes a convolutional neural network; and the encoder is formed by stacking multiple layers of Conformer structures.

[0038] Optionally, the method is applied to a client, and the speech unit prediction model, the acoustic model and the vocoder are deployed from a server to the client;

[0039] Wherein, before deploying the speech unit prediction model, the acoustic model and the vocoder to the client, the server is further used to compress at least one of the speech unit prediction model, the acoustic model and the vocoder according to the computing resources of the client;

[0040] The compression process includes at least one of pruning and quantization; the pruning process indicates deleting redundant parameters or redundant networks in the neural network model; the quantization process indicates reducing the number of bits of parameters and / or activation values ​​in the neural network model.

[0041] According to a fourth aspect of an embodiment of the present disclosure, a server is provided, wherein the server is used to execute the method described in any one of the first aspect, the second aspect or the third aspect.

[0042] According to a fifth aspect of an embodiment of the present disclosure, a client is provided, wherein the client is used to execute the method described in any one of the third aspects.

[0043] According to a sixth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0044] processor;

[0045] a memory for storing processor-executable instructions;

[0046] When the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0047] According to a seventh aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a structural diagram of an interactive system provided by an exemplary embodiment.

[0050] Figure 2 It is a flowchart of a method for training a speech unit prediction model provided by an exemplary embodiment.

[0051] Figure 3 It is a schematic diagram of a training process of a speech unit prediction model provided by an exemplary embodiment.

[0052] Figure 4 It is a flowchart of another method for training a speech unit prediction model provided by an exemplary embodiment.

[0053] Figure 5 It is a schematic diagram of a training process of another speech unit prediction model provided by an exemplary embodiment.

[0054] Figure 6 It is a schematic diagram of a speech synthesis process provided by an exemplary embodiment.

[0055] Figure 7 It is a structural schematic diagram of an acoustic model provided by an exemplary embodiment.

[0056] Figure 8 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0058] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0059] Here is an explanation of the relevant terms:

[0060] Phonetic unit: includes the unit of sound produced when speaking, which can be phonemes, syllables, phonemes, etc.

[0061] A phone is a speech unit divided according to the natural properties of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phone. Phones are divided into two major categories: vowels and consonants. For example, the Chinese syllable ā (ah) has only one phone, ài (love) has two phones, dāi (stupid) has three phones, etc.; English has 48 phones, including 20 vowels (such as / i: / , / u: / , / a: / , etc.) and 28 consonants (such as / p / , / t / , / k / , etc.).

[0062] A syllable is a speech unit formed by the combination of phones (including consonants and vowels) for pronunciation. The syllables in alphabetic languages are pronounced by the combination of vowels and consonants, and the syllables in Chinese are pronounced by the combination of initials and finals. A single vowel that can be pronounced is also a syllable.

[0063] A phoneme is a speech unit that can distinguish meanings in a language or dialect.

[0064] In related technologies, text-to-speech (TTS) technology is a technology that converts the text information generated by a computer itself or input externally into understandable and fluent spoken language output; it involves multiple disciplinary technologies such as acoustics, linguistics, digital signal processing, and computer science. It is a cutting-edge technology in the field of information processing. The main problem it solves is how to convert text information into audible sound information, that is, to make the machine speak like a human.

[0065] The speech synthesis process first converts the text into a sequence of speech units, and then generates a speech waveform according to the sequence of speech units. Among them, the process of converting the text into a sequence of speech units in the first step involves linguistic processing, which generally needs to be realized through the entire link of word segmentation, pronunciation conversion, text regularization, prosody prediction, and polyphone prediction; the second step requires speech synthesis technology to synthesize high-quality speech streams. Among them, the process of converting the text into a sequence of speech units in the speech synthesis process is relatively complex and involves multiple links. A too long processing chain will lead to the problem of error accumulation, which will make the synthesis prone to errors.

[0066] In response to the problems in related technologies, the embodiments of this specification implement training a speech unit prediction model. Through this speech unit prediction model, the text can be converted into a sequence of speech units at one time, without going through multiple links such as word segmentation, pronunciation conversion, text regularization, prosody prediction, and polyphone prediction, realizing the simplification of the process of converting text into a sequence of speech units, which is beneficial to reducing error accumulation, improving the success rate of speech synthesis, and at the same time, the one-time conversion process is also beneficial to improving the efficiency of speech synthesis.

[0067] Furthermore, considering that in the supervised training process, too many training samples will lead to high manual labeling costs, and too few training samples will lead to poor model training results.

[0068] In view of the above problems, the embodiment of this specification provides a training method for a speech unit prediction model, which can obtain a first training sample and a second training sample; the first training sample includes a number of texts, and the second training sample includes a number of texts and their speech units; the number of texts in the first training sample is greater than the number of texts in the second training sample; the first training sample can be used to perform self-supervised pre-training on a preset neural network model to obtain a language model; and then the second training sample is used to perform supervised training on the language model to obtain a speech unit prediction model. In this embodiment, the neural network model can be pre-trained by self-supervised learning. The self-supervised learning process does not require manual annotation, so a large amount of text can be obtained for pre-training, so that the neural network model can fully learn the language expression information and obtain the language model; on this basis, only an appropriate number of second training samples need to be used to perform supervised training on the language model, so that the language model that has learned the language expression information can further learn the mapping relationship between text and speech units, and finally obtain the speech unit prediction model. In the case of a small number of second training samples, the prediction accuracy of the speech unit prediction model can also be improved by the above method, and it is also beneficial to reduce the annotation cost.

[0069] In response to the above problems, the embodiments of the present specification also provide another method for training a speech unit prediction model, which can obtain a first training sample, a second training sample and a first neural network model; the first training sample includes a number of texts, the second training sample includes a number of texts and their speech unit labels, and the number of texts in the first training sample is greater than the number of texts in the second training sample; the first neural network model learns language expression information through a self-supervised pre-training process; the preset second neural network model can be trained for knowledge distillation based on the first training sample and the first neural network model to obtain a language model that learns language expression information from the first neural network model; the scale of the language model is smaller than the scale of the first neural network model; and then the language model is supervised trained using the second training sample to obtain a speech unit prediction model. This embodiment takes into account that in the related art there may be a first neural network model that has learned language expression information but may not be suitable for application scenarios of the speech unit prediction model. Therefore, the language expression information can be transferred from the large-scale first neural network model to the small-scale language model through knowledge distillation to adapt to the application scenarios of the speech unit prediction model. On this basis, it is only necessary to use an appropriate number of second training samples to conduct supervised training on the language model, so that the language model that has learned the language expression information can further learn the mapping relationship between text and speech units, and finally obtain the speech unit prediction model. When the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, which is also beneficial to reduce the annotation cost.

[0070] See also Figure 1 , Figure 1 The following is an exemplary structural diagram of an interactive system. The interactive system includes a server 10 and a client 20. The server 10 and the client 20 can be connected via a network, such as the client 20 can be connected to the Internet via GPRS\4G\5G\WIFI and the like.

[0071] The server 10 may be a server 10 deployed on a cloud server, or a server 10 deployed on a server dedicated to building, distributing, and managing speech synthesis.

[0072] In different interaction scenarios, the electronic device where the client 20 is located is also different. The electronic device includes but is not limited to: voice navigation equipment, story machines, smart speakers, dictionary pens, wearable devices (for example, watches, glasses, gloves, headwear (for example, hats, helmets, virtual reality headsets, augmented reality headsets, head-mounted devices (HMD), headbands), pendants, armbands, leg rings, shoes, vests), and also includes mobile communication equipment, namely: commonly known as mobile phones or smart phones, as well as personal computers, PADs, iPads and other terminal devices.

[0073] The training method of the speech unit prediction model provided in the embodiment of the present disclosure can be executed by the server, and the trained speech unit prediction model is used to process the input text to obtain a speech unit sequence, which can be applied to speech synthesis scenarios. Among them, if the speech synthesis process is carried out in the server, the trained speech unit prediction model can be deployed in the server; if the speech synthesis process is carried out in the client, the trained speech unit prediction model can be deployed in the client; of course, in addition to deploying the speech unit prediction model, the speech synthesis process also needs to deploy an acoustic model and a vocoder, the acoustic model is used to convert the speech unit sequence into acoustic features, and the vocoder is used to convert the acoustic features into speech data.

[0074] Next, the training method of the speech unit prediction model is explained:

[0075] See also Figure 2 as well as Figure 3 , Figure 2 A flowchart of a method for training a speech unit prediction model provided in an embodiment of this specification. The method can be executed by a server, and the method includes:

[0076] In S201, a first training sample and a second training sample are obtained; wherein the first training sample includes a number of texts, and the second training sample includes a number of texts and their speech units; and the number of texts in the first training sample is greater than the number of texts in the second training sample.

[0077] In S202, the first training sample is used to perform self-supervised pre-training on a preset neural network model to obtain a language model that learns language expression information from the first training sample.

[0078] In S203, supervised training is performed on the language model using the second training sample to obtain a speech unit prediction model; wherein the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0079] In this embodiment, the neural network model can be pre-trained by self-supervised learning. The self-supervised learning process does not require manual annotation, so a large amount of text can be obtained for pre-training, so that the neural network model can fully learn the language expression information and obtain a language model; on this basis, it is only necessary to use an appropriate number of second training samples to conduct supervised training on the language model, so that the language model that has learned the language expression information can further learn the mapping relationship between text and speech units, and finally obtain a speech unit prediction model. When the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, which is also beneficial to reduce the annotation cost.

[0080] Exemplarily, the several texts in the first training sample include texts obtained from the Internet using crawler technology and texts obtained from the developer's internal business data; among them, the texts obtained from the Internet are far more than the texts obtained from the developer's internal business data. These two parts of data (which can reach hundreds of millions) are combined for self-supervised learning, so that the language model can fully learn language expression information from massive texts, and the language expression information includes but is not limited to semantic information, syntactic information, and contextual relationships.

[0081] Exemplarily, the plurality of texts in the second training sample include texts obtained from the developer's internal business data (may involve tens of thousands of texts), and only a suitable amount of text needs to be annotated, which helps to reduce the annotation cost.

[0082] In some embodiments, the pre-training process is exemplified here: the server can replace the text at a random position in the first training sample with a mask symbol to obtain the masked text. Then, the masked text is used as input and the pre-mask text is used as a label, and the following process is executed in a loop until the loop end condition is met: the current neural network model is used to predict the text at the mask position in the masked text to obtain the predicted text; if the loop end condition is not met, the parameters of the current neural network model are adjusted according to the predicted text and the pre-mask text to obtain the adjusted neural network model; wherein the adjusted neural network model is used as the current neural network model for the next loop process.

[0083] Exemplarily, the loss function involved in the training process may include a mask reconstruction loss function, the goal of which is to enable the neural network model to predict the masked input based on context information. When calculating the mask reconstruction loss function, the reconstruction error can be calculated based on the difference between the predicted text and the text before the mask, for example, using a cross-entropy loss function or a mean squared error loss function.

[0084] In addition to the mask reconstruction loss function, other self-supervised objective functions can be combined to further improve the pre-training effect, such as the context continuity loss function. Different tasks and data sets may require different loss function choices, and the specific choice should be adjusted according to the actual situation and pre-training goals.

[0085] In self-supervised pre-training, the input text is usually divided into a series of segments or windows, each containing a certain number of words. The contextual continuity loss function uses the relationship between these segments to provide additional supervision signals. Specifically, for each input segment, the model predicts the next word or character of the segment based on the context information. The continuity loss is then calculated using the difference between the predicted result and the actual next word or character. Such a loss encourages the model to accurately predict words or characters in the context and to capture local continuity features in the input text. The continuity loss can be calculated using the cross entropy loss function.

[0086] The loop end condition includes but is not limited to the loop number reaching a preset number, or the difference between the predicted text and the text before the mask is less than a preset threshold.

[0087] When the loop end condition is met, the current neural network model in the last loop process is used as the language model. In this embodiment, during the pre-training process, the neural network model learns the representation of the input text by learning the context relationship, and by predicting the words or characters at the mask position, the neural network model gradually learns to understand the semantic and syntactic information and can generate a reasonable representation.

[0088] Exemplarily, the language model includes but is not limited to at least one of a Transformer model, a BERT (Bidirectional Encoder Representations from Transformers) model, and a GPT (Generative Pre-trained Transformer) model. The Transformer model is a neural network model based on a self-attention mechanism, which can be used for natural language processing and other sequence-to-sequence tasks. The BERT model is a pre-trained natural language processing model. The GPT model is a pre-trained generative language model.

[0089] In some embodiments, after obtaining the language model, the server may use the second training sample to perform supervised training on the language model to obtain a speech unit prediction model. The speech unit prediction model is used to process the input text to obtain a speech unit sequence. For example, the input text is "Hello, how is the weather today?", and the speech unit sequence output by the speech unit prediction model is "nih ao j in ti an ti an qi z en m ey ang".

[0090] Supervised learning is a machine learning task that infers functions from labeled training data sets. Samples with known certain characteristics are used as training sets to establish a mathematical model (such as the discriminant model in pattern recognition, the weight model in artificial neural network method, etc.), and then the established model is used to predict unknown samples.

[0091] Here is an exemplary explanation of the supervised training process: taking the text as input and the speech unit corresponding to the text as a label, the following process is executed in a loop until the loop end condition is met: using the current language model to predict the predicted speech unit corresponding to the text; if the loop end condition is not met, adjusting the parameters of the current language model according to the predicted speech unit and the speech unit label to obtain the adjusted language model; wherein the adjusted language model is used as the current language model for the next loop process. During the training process, the server can calculate the loss function according to the predicted speech unit and the speech unit label, and then reversely adjust the parameters of the current language model according to the calculated loss value. The loss function includes, but is not limited to, the mean square error loss function, the cross entropy loss function, or the mean absolute error loss function, etc., and this embodiment does not impose any restrictions on this.

[0092] Among them, the loop end conditions in the supervised training process include but are not limited to the number of loops reaching a preset number of times, or the difference between the predicted speech unit and the speech unit label is less than a preset threshold. When the loop end conditions are met, the server can use the current language model in the last round of loops as the speech unit prediction model. This embodiment allows the language model that has learned the language expression information to further learn the mapping relationship between text and speech units, and finally obtains the speech unit prediction model. Since the language model has learned the language expression information in advance, the prediction accuracy of the speech unit prediction model can be improved by the above method when the second training sample is small, and it is also beneficial to reduce the annotation cost.

[0093] In some embodiments, see Figure 1 ,The trained speech unit prediction model will be deployed on the server or client according to different ,speech synthesis requirements, and the computing resources of different ,electronic devices are also different.

[0094] If the trained speech unit prediction model is deployed on the server, then in order to adapt to the computing resources of the server, before pre-training, the scale of the language model can be pre-determined according to the computing resources of the server, and the scale of the language model is positively correlated with the computing resources of the client or the server. The more computing resources, the larger the scale of the language model can be set to improve the accuracy of the model output. The fewer computing resources, the smaller the scale of the language model can be set to increase the speed of the model output, so that the finally trained speech unit prediction model meets the needs of the server.

[0095] If the trained speech unit prediction model is deployed in the client, then in order to adapt to the computing resources of the client, before pre-training, the scale of the language model can be pre-determined according to the computing resources of the client, and the scale of the language model is positively correlated with the computing resources of the client or the server, so that the final trained speech unit prediction model meets the needs of the client.

[0096] The scale of the language model reflects information such as the number of parameters and the number of layers of the language model. The larger the number of parameters, the larger the scale of the language model, and / or the more layers, the larger the scale of the language model.

[0097] See also Figure 4 as well as Figure 5 , Figure 4 A flowchart of a method for training a speech unit prediction model provided in an embodiment of this specification. The method can be executed by a server, and the method includes:

[0098] In S401, a first training sample, a second training sample and a first neural network model are obtained; wherein the first training sample includes a plurality of texts, the second training sample includes a plurality of texts and their speech units, and the number of texts in the first training sample is greater than the number of texts in the second training sample; and the first neural network model learns language expression information through a self-supervised pre-training process.

[0099] In S402, knowledge distillation training is performed on a preset second neural network model based on the first training sample and the first neural network model to obtain a language model that learns language expression information from the first neural network model; the scale of the language model is smaller than the scale of the first neural network model.

[0100] In S403, supervised training is performed on the language model using the second training sample to obtain a speech unit prediction model; wherein the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0101] This embodiment takes into account that there may be a first neural network model that has learned language expression information in the related technology, but it may not be suitable for the application scenario of the speech unit prediction model due to reasons such as its large scale. Therefore, the language expression information can be transferred from the large-scale first neural network model to the small-scale language model through the knowledge distillation process to adapt to the application scenario of the speech unit prediction model; on this basis, it is only necessary to use an appropriate number of second training samples to perform supervised training on the language model, so that the language model that has learned the language expression information can further learn the mapping relationship between text and speech units, and finally obtain the speech unit prediction model. When the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, which is also beneficial to reduce the annotation cost.

[0102] Exemplarily, the several texts in the first training sample include texts obtained from the Internet using crawler technology and texts obtained from the developer's internal business data; wherein the texts obtained from the Internet are far more than the texts obtained from the developer's internal business data. The first training sample contains at least a portion of text used to train the first neural network model. For example, the text used to train the first neural network model can be obtained from the Internet using crawler technology so that the language model can learn the knowledge of the first neural network model. At the same time, the first training sample contains texts obtained from the developer's internal business data so that the language model can also learn language expression information that meets the developer's business needs.

[0103] Exemplarily, the plurality of texts in the second training sample include texts obtained from the developer's internal business data (possibly involving tens of thousands of texts), which are used in a subsequent supervised training process. Only an appropriate amount of text needs to be annotated, which helps to reduce the annotation cost.

[0104] The first neural network model may be a pre-trained model obtained from the Internet or other means. For example, the first neural network model may be a model that has learned language expression information through a self-supervised pre-training process. The specific pre-training process can be found in the above description and will not be repeated here. In order to improve the accuracy of the model output results, the first neural network model may be a relatively large-scale model, such as a large number of parameters and / or layers, which may not be suitable for the application scenario of the speech unit prediction model. For example, when the speech unit prediction model is deployed on the client, the client's computing resources are limited and it is impossible to run a large-scale speech unit prediction model. If the first neural network model is directly used for supervised training, the scale of the speech unit prediction model obtained is also relatively large, and it is obviously impossible to deploy it to the client. For another example, when the speech unit prediction model is deployed on the server, the computing resources of the server running the speech unit prediction model and the computing resources of the computing device running the first neural network model may also be different, resulting in the inability to directly reuse the first neural network model.

[0105] Therefore, the embodiments of the present specification implement knowledge distillation training of a preset second neural network model based on the first training samples and the first neural network model to obtain a language model that learns language expression information from the first neural network model; the scale of the language model is smaller than the scale of the first neural network model, and the second training samples can be used to perform supervised training on the language model to obtain a speech unit prediction model that meets the requirements.

[0106] In some embodiments, see Figure 1 ,The trained speech unit prediction model will be deployed on the server or client according to different ,speech synthesis requirements, and the computing resources of different ,electronic devices are also different.

[0107] If the trained speech unit prediction model is deployed on the server, in order to adapt to the computing resources of the server, before conducting knowledge distillation training, the scale of the language model can be pre-determined according to the computing resources of the server, and the scale of the language model (or the second neural network model) is positively correlated with the computing resources of the client or the server. The more computing resources, the larger the scale of the language model can be set to improve the accuracy of the model output. The fewer computing resources, the smaller the scale of the language model can be set to increase the speed of the model output, so that the finally trained speech unit prediction model meets the needs of the server.

[0108] If the trained speech unit prediction model is deployed in the client, in order to adapt to the computing resources of the client, before conducting knowledge distillation training, the scale of the language model can be pre-determined according to the computing resources of the client, and the scale of the language model (or the second neural network model) is positively correlated with the computing resources of the client or the server, so that the finally trained speech unit prediction model meets the needs of the client.

[0109] The scale of the language model reflects information such as the number of parameters and the number of layers of the language model. The larger the number of parameters, the larger the scale of the language model, and / or the more layers, the larger the scale of the language model.

[0110] After determining the size of the language model (or the second neural network model), the knowledge distillation process can be performed. Knowledge distillation improves the performance of a small model by transferring the knowledge of a complex large model to a simplified small model. This process is achieved by using the output of the large model as an additional training target and using it together with the original labels to train the small model. Through knowledge distillation, the small model is able to obtain richer knowledge and representation capabilities from the large model, and has the ability to approach or even exceed the performance of the large model while maintaining relatively low complexity. This is very useful for deploying models in resource-constrained environments or improving the efficiency of models on edge devices.

[0111] Here is an exemplary explanation of the process of obtaining a language model: the server can replace the text at a random position in the first training sample with a mask symbol to obtain the masked text. Then the following steps are executed in a loop until the loop end condition is met: use the current second neural network model to predict the text at the mask position in the masked text to obtain the first predicted text; and use the first neural network model to predict the text at the mask position in the masked text to obtain the second predicted text; if the loop end condition is not met, adjust the parameters of the current second neural network model according to the first predicted text, the second predicted text and the text before masking to obtain the adjusted second neural network model; wherein the adjusted second neural network model is used as the current second neural network model for the next loop process.

[0112] During the loop, a distillation loss function is used to measure the difference between the prediction of the second neural network model and the output of the first neural network model. The distillation loss function can be calculated based on the output probability distribution of the first neural network model and the output probability distribution of the second neural network model. As the training progresses, the optimization of the distillation loss function will gradually make the second neural network model approach the prediction result of the first neural network model. The distillation loss function is used for the difference between the prediction of the second neural network model and the output of the first neural network model. At the same time, the difference between the prediction of the second neural network model and the original text label needs to be considered. Therefore, the loss can be calculated based on the first predicted text and the second predicted text, and the loss can be calculated based on the first predicted text and the text before the mask. Then the two losses can be weighted and combined, and the weights can be determined according to actual requirements.

[0113] The loop end condition includes but is not limited to the loop number reaching a preset number, or the difference between the predicted text and the text before the mask is less than a preset threshold.

[0114] When the loop end condition is met, the current second neural network model in the last loop process can be used as the language model. In this embodiment, through knowledge distillation, the second neural network model can obtain richer knowledge and representation capabilities from the first neural network model, and has the ability to approach or even exceed the performance of the first neural network model while maintaining a relatively low complexity. Since knowledge transfer is achieved on the basis of the pre-trained first neural network model, compared to Figure 2 The pre-training process from scratch in the embodiment can obtain a language model more quickly.

[0115] Exemplarily, the language model includes but is not limited to at least one of a Transformer model, a BERT (Bidirectional Encoder Representations from Transformers) model, and a GPT (Generative Pre-trained Transformer) model. The Transformer model is a neural network model based on a self-attention mechanism, which can be used for natural language processing and other sequence-to-sequence tasks. The BERT model is a pre-trained natural language processing model. The GPT model is a pre-trained generative language model.

[0116] In some embodiments, after acquiring the language model, the server may use the second training sample to perform supervised training on the language model to obtain a speech unit prediction model. The speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0117] Here is an exemplary explanation of the supervised training process: taking the text as input and the speech unit corresponding to the text as a label, the following process is executed in a loop until the loop end condition is met: using the current language model to predict the predicted speech unit corresponding to the text; if the loop end condition is not met, adjusting the parameters of the current language model according to the predicted speech unit and the speech unit label to obtain the adjusted language model; wherein the adjusted language model is used as the current language model for the next loop process. During the training process, the server can calculate the loss function according to the predicted speech unit and the speech unit label, and then reversely adjust the parameters of the current language model according to the calculated loss value. The loss function includes, but is not limited to, the mean square error loss function, the cross entropy loss function, or the mean absolute error loss function, etc., and this embodiment does not impose any restrictions on this.

[0118] Among them, the loop end conditions in the supervised training process include but are not limited to the number of loops reaching a preset number of times, or the difference between the predicted speech unit and the speech unit label is less than a preset threshold. When the loop end conditions are met, the server can use the current language model in the last round of loops as the speech unit prediction model. This embodiment allows the language model that has learned the language expression information to further learn the mapping relationship between text and speech units, and finally obtains the speech unit prediction model. Since the language model has learned the language expression information in advance, the prediction accuracy of the speech unit prediction model can be improved by the above method when the second training sample is small, and it is also beneficial to reduce the annotation cost.

[0119] In some embodiments, the application process of the speech unit prediction model is exemplarily described here: Figure 6 The embodiment of this specification provides a speech synthesis method, which can be executed by a server or a client, and the method includes:

[0120] In S601, the text to be processed is obtained.

[0121] In S602, the text to be processed is converted into a speech unit sequence using a speech unit prediction model; wherein the speech unit prediction model is trained based on any one of the training methods described above.

[0122] In S603, the speech unit sequence is processed using an acoustic model to obtain acoustic features.

[0123] In S604, the acoustic features are converted into speech data using a vocoder.

[0124] This embodiment applies the speech unit prediction model obtained through the above training in the speech synthesis process. The speech unit prediction model can be used to convert text into a speech unit sequence at one time without going through multiple steps such as word segmentation, character-phonetic conversion, text regularization, rhythm prediction and polyphonetic character prediction. This simplifies the process of converting text into a speech unit sequence, which is beneficial to reducing error accumulation and improving the success rate of speech synthesis. At the same time, the one-time conversion process is also beneficial to improving the efficiency of speech synthesis.

[0125] In some embodiments, if the speech synthesis method is executed by a server, the speech unit prediction model, the acoustic model and the vocoder are deployed on the server. After acquiring the speech data, the server can send it to the client so that the speech data can be played on the client.

[0126] In other embodiments, if the speech synthesis method is executed by a client, the speech unit prediction model, the acoustic model and the vocoder are deployed on the client, and the client can directly play the speech data after acquiring the speech data.

[0127] In some embodiments, the training process of the acoustic model and the vocoder is exemplified here: first, a speaker's corpus is constructed, which consists of speech and its corresponding text, the speech is converted into acoustic features, and the text is converted into a sequence of speech units, and through supervised learning, the acoustic model learns the mapping relationship between the speech unit sequence and the acoustic features, and the vocoder learns the mapping relationship between the acoustic features and the speech.

[0128] Exemplarily, considering that the speech synthesis method of the present specification can be applied to different scenarios, such as news broadcasting scenarios, storytelling scenarios, voice navigation scenarios, etc., and considering that different scenarios have different language expressions, multiple speaker libraries can be constructed during the training phase for supervised training, including but not limited to news speaker libraries, navigation speaker libraries, storytelling speaker libraries, etc., so that the trained acoustic models and acoustic models can be applicable to different scenarios.

[0129] In some embodiments, see Figure 7 The acoustic model includes a decoder, a parameter adjuster and an encoder; the speech unit sequence is a discrete character, and the decoder is used to process the speech unit sequence to obtain a more compact representation vector representing contextual semantic information; the parameter adjuster is used to process the representation vector to obtain speech information, and the speech information includes at least one of the duration, volume and pitch of the speech; the decoder is used to process the representation vector and the speech information to obtain acoustic features.

[0130] Exemplarily, the encoder is composed of multiple layers of Transformer components stacked together; the parameter adjuster includes a convolutional neural network; and the encoder is composed of multiple layers of Conformer structures stacked together.

[0131] In some embodiments, the vocoder may be a user-specific vocoder or a vocoder common to multiple users. The vocoder may be an LPCNet vocoder, a WaveRNN vocoder, a HiFi-gan vocoder, etc., without any limitation.

[0132] In some embodiments, when the speech synthesis method is applied to a client, the server may deploy the speech unit prediction model, the acoustic model, and the vocoder to the client. In order to adapt to the computing resources of the client, the server is further used to compress at least one of the speech unit prediction model, the acoustic model, and the vocoder according to the computing resources of the client before deploying the speech unit prediction model, the acoustic model, and the vocoder to the client.

[0133] Exemplarily, the compression processing includes at least one of pruning processing and quantization processing; the pruning processing indicates deleting redundant parameters or redundant networks in the neural network model; the quantization processing indicates reducing the number of bits of parameters and / or activation values ​​in the neural network model.

[0134] Among them, pruning processing includes parameter pruning and network pruning.

[0135] Parameter pruning refers to reducing the number of parameters in a model by pruning redundant or unimportant parameters in the model. Common pruning methods include structured pruning and unstructured pruning. Taking the parameter pruning of the speech unit prediction model as an example, first, the importance of the parameters of the speech unit prediction model can be evaluated to determine the importance of each parameter. Common parameter importance evaluation methods include structured pruning methods (such as L1 regularization) and unstructured pruning methods (such as sensitivity analysis, Taylor approximation, etc.). Then, according to the evaluation results of the importance of the parameters, a certain proportion of parameters are selected for pruning; the pruning method can be to directly set the parameters to zero (sparse pruning) or delete unimportant parameters; the pruned model will become more sparse, that is, it contains more zero parameters. Finally, the pruned model can be fine-tuned, that is, the pruned model is retrained using the second training sample mentioned above to restore or improve its performance. The purpose of fine-tuning is to minimize performance loss while maintaining the sparsity of the model.

[0136] Network pruning is similar to parameter pruning, but it is not just about pruning weights, it also includes pruning neurons, pruning layers, etc. By pruning the entire network structure, the size and computational complexity of the model can be reduced.

[0137] Quantization refers to converting the parameters and activation values ​​in the neural network model from floating point numbers to fixed point numbers or floating point numbers with lower precision. Take the quantization of the parameters of the speech unit prediction model as an example. For example, the floating point parameters of the speech unit prediction model can be mapped to integers or fixed point decimals with a fixed number of bits, and then restored to floating point numbers using a scaling factor. After the quantization, the quantized model can be fine-tuned, that is, the quantized model can be retrained using the second training sample mentioned above to restore or improve its performance. The purpose of fine-tuning is to minimize performance loss while maintaining model quantization.

[0138] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0139] In some embodiments, see Figure 1 The embodiment of this specification also provides a server, which can execute any of the methods described above.

[0140] Exemplarily, the speech unit prediction model may be trained in the server, and the speech unit prediction model, the acoustic model, and the vocoder may be deployed in the server so that the server performs the speech synthesis process.

[0141] Of course, the training process of the speech unit prediction model can also be performed by other computing devices different from the server, and this embodiment does not impose any restrictions on this.

[0142] In some embodiments, see Figure 1 The embodiment of this specification also provides a client, which can execute the above-mentioned speech synthesis method.

[0143] Exemplarily, the speech unit prediction model, the acoustic model, and the vocoder may be deployed in the client so that the client performs the speech synthesis process.

[0144] In some embodiments, the embodiments of this specification also provide an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0145] For example, Figure 8 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 8 At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into the memory 808 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0146] In some embodiments, the embodiments of this specification further provide a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the steps of any of the methods described above are implemented.

[0147] In some embodiments, the embodiments of this specification further provide a computer program product, which implements the steps of any of the above methods when executed by a processor.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0149] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0150] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0152] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0154] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0155] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0156] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0157] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for training a speech unit prediction model, characterized in that: include: Obtain a first training sample and a second training sample; wherein the first training sample includes a plurality of texts, and the second training sample includes a plurality of texts and their speech units; the number of texts in the first training sample is greater than the number of texts in the second training sample; Using the first training sample to perform self-supervised pre-training on a preset neural network model to obtain a language model that learns language expression information from the first training sample; The language model is supervisedly trained using the second training sample to obtain a speech unit prediction model; wherein the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

2. The method according to claim 1, characterized in that The using the first training sample to perform self-supervised pre-training on a preset neural network model includes: Replacing the text at random positions in the first training sample with mask symbols to obtain masked text; Using the masked text as input and the pre-masked text as a label, the following process is executed in a loop until the loop end condition is met: using the current neural network model to predict the text at the mask position in the masked text to obtain the predicted text; if the loop end condition is not met, adjusting the parameters of the current neural network model according to the predicted text and the pre-masked text to obtain an adjusted neural network model; wherein the adjusted neural network model is used as the current neural network model of the next loop process; When the loop end condition is met, the current neural network model in the last loop process is used as the language model.

3. A method for training a speech unit prediction model, characterized in that: include: Obtain a first training sample, a second training sample and a first neural network model; wherein the first training sample includes a plurality of texts, the second training sample includes a plurality of texts and their speech units, and the number of texts in the first training sample is greater than the number of texts in the second training sample; the first neural network model learns language expression information through a self-supervised pre-training process; Performing knowledge distillation training on a preset second neural network model based on the first training sample and the first neural network model to obtain a language model that learns language expression information from the first neural network model; the scale of the language model is smaller than the scale of the first neural network model; The language model is supervisedly trained using the second training sample to obtain a speech unit prediction model; wherein the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

4. The method according to claim 3, characterized in that The first training sample at least includes a portion of text used to train the first neural network model; The step of performing knowledge distillation training on a preset second neural network model based on the first training sample pair and the first neural network model to obtain a language model that learns language expression information from the neural network model includes: Replacing the text at random positions in the first training sample with mask symbols to obtain masked text; The following steps are executed in a loop until the loop end condition is met: using the current second neural network model to predict the text at the mask position in the masked text to obtain the first predicted text; and using the first neural network model to predict the text at the mask position in the masked text to obtain the second predicted text; if the loop end condition is not met, adjusting the parameters of the current second neural network model according to the first predicted text, the second predicted text and the text before masking to obtain the adjusted second neural network model; wherein the adjusted second neural network model is used as the current second neural network model of the next loop process; When the loop end condition is met, the current second neural network model in the last loop process is used as the language model.

5. The method according to any one of claims 1 to 4, characterized in that: The speech unit prediction model is used to be deployed to a client or a server; The scale of the language model is determined according to the computing resources of the client or the server, and the scale of the language model is positively correlated with the computing resources of the client or the server.

6. The method according to any one of claims 1 to 4, characterized in that: The language model includes at least one of a Transformer model, a BERT model, and a GPT model; and / or The plurality of texts in the first training sample include texts obtained from the Internet using crawler technology and texts obtained from the developer's internal business data; the plurality of texts in the second training sample include texts obtained from the developer's internal business data.

7. A speech synthesis method, characterized in that: include: Get the text to be processed; The text to be processed is converted into a speech unit sequence using a speech unit prediction model; wherein the speech unit prediction model is trained based on the training method according to any one of claims 1 to 6; Processing the speech unit sequence using an acoustic model to obtain acoustic features; The acoustic features are converted into speech data using a vocoder.

8. The method according to claim 7, characterized in that The acoustic model includes a decoder, a parameter adjuster and an encoder; The decoder is used to process the speech unit sequence to obtain a representation vector representing contextual semantic information; The parameter adjuster is used to process the representation vector to obtain speech information, wherein the speech information includes at least one of the duration, volume and pitch of the speech; The decoder is used to process the representation vector and the speech information to obtain acoustic features.

9. The method according to claim 7 or 8, characterized in that: The method is applied to a client, and the speech unit prediction model, the acoustic model and the vocoder are deployed from a server to the client; Wherein, before deploying the speech unit prediction model, the acoustic model and the vocoder to the client, the server is further used to compress at least one of the speech unit prediction model, the acoustic model and the vocoder according to the computing resources of the client; The compression process includes at least one of pruning and quantization; the pruning process indicates deleting redundant parameters or redundant networks in the neural network model; the quantization process indicates reducing the number of bits of parameters and / or activation values ​​in the neural network model.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

Images