Speech unit prediction model training method, speech synthesis method, and electronic device

Abstract

The one or more embodiments of the specification provide a speech unit prediction model training method, a speech synthesis method and an electronic device. The training method comprises: obtaining a first training sample and a second training sample; wherein the first training sample comprises a plurality of texts, and the second training sample comprises a plurality of texts and speech units thereof; the number of texts in the first training sample is greater than the number of texts in the second training sample; a preset neural network model is pre-trained in a self-supervised manner by using the first training sample, to obtain a language model learning language expression information from the first training sample; the language model is supervised trained by using the second training sample, to obtain a speech unit prediction model; and the speech unit prediction model is used for processing input texts to obtain a speech unit sequence. In this embodiment, the language model learning language expression information is obtained by pre-training, and then the language model is supervised trained, which is beneficial to improve the prediction accuracy of the speech unit prediction model.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of terminal technology, and in particular to a training method for a speech unit prediction model, a speech synthesis method, and an electronic device. Background Technology

[0002] With its rapid development in recent years, voice interaction technology has gradually entered people's daily lives. Among them, speech synthesis technology, as an important component of voice interaction technology, plays a vital role in the voice interaction process.

[0003] Text-to-speech (TTS) technology is used to transform computer-generated or externally input text information into fluent, understandable spoken output. The TTS process first converts text into a sequence of speech units, and then generates a speech waveform based on this sequence. The first step, converting text into a sequence of speech units, involves linguistic processing, typically requiring word segmentation, phonetic conversion, text regularization, prosodic prediction, and polyphonic character prediction—a complete chain of steps. The second step requires TTS technology to synthesize a high-quality speech stream. The process of converting text into a sequence of speech units is quite complex, involving multiple stages. An excessively long processing chain can lead to error accumulation, making the synthesis prone to errors and degrading sound quality. Summary of the Invention

[0004] In view of the above, this specification provides one or more embodiments of a training method for a speech unit prediction model, a speech synthesis method, a server, a client, an electronic device, and a computer-readable storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for training a speech unit prediction model is proposed, comprising:

[0007] Obtain a first training sample and a second training sample; wherein, the first training sample includes several texts, and the second training sample includes several texts and their speech units; the number of texts in the first training sample is greater than the number of texts in the second training sample;

[0008] The first training sample is used to perform self-supervised pre-training on a preset neural network model to obtain a language model that has learned language expression information from the first training sample.

[0009] The language model is trained in a supervised manner using the second training sample to obtain a speech unit prediction model; wherein the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0010] Beneficial effects: In this embodiment, self-supervised learning can be used to pre-train the neural network model. The self-supervised learning process does not require manual annotation, so a large amount of text can be obtained for pre-training, allowing the neural network model to fully learn the language expression information and obtain a language model. On this basis, only an appropriate number of second training samples are needed to perform supervised training on the language model, allowing the language model that has learned the language expression information to further learn the mapping relationship between text and speech units, and finally obtain a speech unit prediction model. Even when the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, while also helping to reduce annotation costs.

[0011] Optionally, the step of performing self-supervised pre-training on a preset neural network model using the first training samples includes:

[0012] The text in the first training sample at random positions is replaced with mask symbols to obtain the masked text.

[0013] Using the masked text as input and the unmasked text as labels, the following process is repeated until the loop termination condition is met: using the current neural network model to predict the text at the masked position in the masked text, obtaining the predicted text; if the loop termination condition is not met, adjusting the parameters of the current neural network model based on the predicted text and the unmasked text, obtaining the adjusted neural network model; wherein, the adjusted neural network model serves as the current neural network model for the next loop process;

[0014] If the loop termination condition is met, the current neural network model in the last loop process is used as the language model.

[0015] According to a second aspect of one or more embodiments of this specification, a method for training a speech unit prediction model is proposed, comprising:

[0016] A first training sample, a second training sample, and a first neural network model are obtained; wherein, the first training sample includes several texts, the second training sample includes several texts and their speech units, the number of texts in the first training sample is greater than the number of texts in the second training sample; the first neural network model learns language expression information through a self-supervised pre-training process;

[0017] Based on the first training samples and the first neural network model, a pre-defined second neural network model is trained by knowledge distillation to obtain a language model that learns language expression information from the first neural network model; the size of the language model is smaller than the size of the first neural network model.

[0018] The language model is trained in a supervised manner using the second training sample to obtain a speech unit prediction model; wherein the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0019] Beneficial effects: This embodiment considers that in related technologies, there may be a first neural network model that has learned language expression information, but due to its large scale, it may not be suitable for the application scenario of the speech unit prediction model. Therefore, the language expression information can be transferred from the large-scale first neural network model to a small-scale language model through the knowledge distillation process to adapt to the application scenario of the speech unit prediction model. On this basis, only an appropriate number of second training samples are needed to perform supervised training on the language model, so that the language model that has learned language expression information can further learn the mapping relationship between text and speech units, and finally obtain the speech unit prediction model. Even when the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, which also helps to reduce the annotation cost.

[0020] Optionally, the first training sample contains at least a portion of text used to train the first neural network model;

[0021] The step of training a preset second neural network model with knowledge distillation based on the first training sample pair and the first neural network model to obtain a language model that has learned linguistic expression information from the neural network model includes:

[0022] The text in the first training sample at random positions is replaced with mask symbols to obtain the masked text.

[0023] The following steps are executed repeatedly until the loop termination condition is met: using the current second neural network model to predict the text at the mask position in the text behind the mask, obtaining the first predicted text; and using the first neural network model to predict the text at the mask position in the text behind the mask, obtaining the second predicted text; if the loop termination condition is not met, the parameters of the current second neural network model are adjusted according to the first predicted text, the second predicted text, and the text before the mask, to obtain the adjusted second neural network model; wherein, the adjusted second neural network model serves as the current second neural network model for the next loop process;

[0024] If the loop termination condition is met, the current second neural network model in the last loop process is used as the language model.

[0025] Optionally, if the speech unit prediction model is deployed to a client or a server, the size of the language model is determined based on the computing resources of the client or the server, and the size of the language model is positively correlated with the computing resources of the client or the server.

[0026] Optionally, the language model includes at least one of the Transformer model, BERT model, and GPT model; and / or the text in the first training sample includes text obtained from the Internet using web crawling technology and text obtained from the developer's internal operating data; the text in the second training sample includes text obtained from the developer's internal operating data.

[0027] According to a third aspect of the present disclosure, a speech synthesis method is provided, comprising:

[0028] Get the text to be processed;

[0029] The text to be processed is converted into a sequence of speech units using a speech unit prediction model; wherein the speech unit prediction model is trained based on the training method described in the first or second aspect.

[0030] The speech unit sequence is processed using an acoustic model to obtain acoustic features;

[0031] The acoustic features are converted into speech data using a vocoder.

[0032] Beneficial effects: This embodiment applies the speech unit prediction model trained above in the speech synthesis process. The speech unit prediction model can convert text into a speech unit sequence in one step without going through multiple steps such as word segmentation, character-to-phonetic conversion, text regularization, prosody prediction and polyphonic character prediction. This simplifies the process of converting text into a speech unit sequence, which helps to reduce error accumulation and improve the success rate of speech synthesis. At the same time, the one-step conversion process also helps to improve the efficiency of speech synthesis.

[0033] Optionally, the acoustic model includes a decoder, a parameter adjuster, and an encoder;

[0034] The decoder is used to process the speech unit sequence to obtain a representation vector that represents contextual semantic information;

[0035] The parameter adjuster is used to process the representation vector to obtain speech information, which includes at least one of the following: speech duration, volume, and pitch.

[0036] The decoder is used to process the representation vector and the speech information to obtain acoustic features.

[0037] Optionally, the encoder is composed of stacked multi-layer Transformer components; the parameter adjuster includes a convolutional neural network; the encoder is composed of stacked multi-layer Conformer structures.

[0038] Optionally, the method is applied to the client, and the speech unit prediction model, the acoustic model, and the vocoder are deployed from the server to the client.

[0039] Before deploying the speech unit prediction model, the acoustic model, and the vocoder to the client, the server is further configured to compress at least one of the speech unit prediction model, the acoustic model, and the vocoder according to the client's computing resources.

[0040] The compression process includes at least one of pruning and quantization; the pruning process indicates the removal of redundant parameters or redundant networks in the neural network model; the quantization process indicates the reduction of the number of bits in the parameters and / or activation values ​​in the neural network model.

[0041] According to a fourth aspect of the present disclosure, a server is provided, the server being configured to perform the method described in any one of the first, second, or third aspects.

[0042] According to a fifth aspect of the present disclosure, a client is provided for performing the method described in any one of the third aspects.

[0043] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising:

[0044] processor;

[0045] Memory used to store processor-executable instructions;

[0046] Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0047] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described above.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of an interactive system provided in an exemplary embodiment.

[0050] Figure 2 This is a schematic flowchart illustrating a training method for a speech unit prediction model provided in an exemplary embodiment.

[0051] Figure 3 This is a schematic diagram illustrating the training process of a speech unit prediction model provided in an exemplary embodiment.

[0052] Figure 4 This is a flowchart illustrating another method for training a speech unit prediction model provided in an exemplary embodiment.

[0053] Figure 5 This is a schematic diagram illustrating the training process of another speech unit prediction model provided in an exemplary embodiment.

[0054] Figure 6 This is a schematic diagram of a speech synthesis process provided in an exemplary embodiment.

[0055] Figure 7 This is a schematic diagram of the structure of an acoustic model provided in an exemplary embodiment.

[0056] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0058] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0059] Here is an explanation of the relevant terminology:

[0060] Phonetic units: These include units of sound produced during speech, such as phonemes, syllables, and phonemes.

[0061] A phone is a speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phone. Phones are divided into two major categories: vowels and consonants. For example, the Chinese syllable ā (ah) has only one phone, ài (love) has two phones, dāi (stupid) has three phones, etc.; English has 48 phones, including 20 vowels (such as / i: / , / u: / , / a: / , etc.) and 28 consonants (such as / p / , / t / , / k / , etc.).

[0062] A syllable is a speech unit formed by the combination of phones (including consonants and vowels) for pronunciation. The syllables in alphabetic languages are pronounced by the combination of vowels and consonants, and the syllables in Chinese are pronounced by the combination of initials and finals. A single vowel that can be pronounced is also a syllable.

[0063] A phoneme is a speech unit that can distinguish meanings in a language or dialect.

[0064] In related technologies, text-to-speech (TTS) technology is a technology that converts the text information generated by a computer itself or input externally into understandable and fluent spoken language output; it involves multiple disciplinary technologies such as acoustics, linguistics, digital signal processing, and computer science. It is a cutting-edge technology in the field of information processing. The main problem it solves is how to convert text information into audible sound information, that is, to make the machine speak like a human.

[0065] The speech synthesis process first converts the text into a sequence of speech units, and then generates a speech waveform based on the sequence of speech units. The first step of converting the text into a sequence of speech units involves linguistic processing, generally requiring the implementation of an entire link including word segmentation, pronunciation conversion, text regularization, prosody prediction, and polyphone prediction; the second step requires speech synthesis technology to synthesize high-quality speech streams. Among them, the process of converting text into a sequence of speech units in the speech synthesis process is relatively complex and requires multiple links. A too long processing chain will lead to the problem of error accumulation, which will make the synthesis prone to errors.

[0066] In response to the problems in related technologies, the embodiments of this specification implement training a speech unit prediction model. Through this speech unit prediction model, the text can be converted into a sequence of speech units at one time, without going through multiple links such as word segmentation, pronunciation conversion, text regularization, prosody prediction, and polyphone prediction, realizing the simplification of the process of converting text into a sequence of speech units, which is beneficial to reducing error accumulation, improving the success rate of speech synthesis, and at the same time, the one-time conversion process is also beneficial to improving the efficiency of speech synthesis.

[0067] Furthermore, considering that in supervised training, too many training samples can lead to high manual annotation costs, while too few training samples can result in poor model training performance.

[0068] To address the aforementioned issues, this specification provides a training method for a speech unit prediction model. This method involves acquiring a first training sample and a second training sample. The first training sample includes several texts, and the second training sample includes several texts and their speech units. The number of texts in the first training sample is greater than the number of texts in the second training sample. The first training sample can be used to perform self-supervised pre-training on a pre-defined neural network model to obtain a language model. Then, the second training sample is used to perform supervised training on the language model to obtain a speech unit prediction model. This embodiment utilizes self-supervised learning to pre-train the neural network model. Since self-supervised learning does not require manual annotation, a large amount of text can be acquired for pre-training, allowing the neural network model to fully learn language expression information and obtain a language model. Based on this, only an appropriate number of second training samples are needed to perform supervised training on the language model, allowing the language model, having learned language expression information, to further learn the mapping relationship between text and speech units, ultimately obtaining a speech unit prediction model. Even with a small number of second training samples, the prediction accuracy of the speech unit prediction model can be improved using the above method, while also reducing annotation costs.

[0069] To address the aforementioned issues, this specification provides another method for training a speech unit prediction model. This method involves acquiring a first training sample, a second training sample, and a first neural network model. The first training sample includes several texts, and the second training sample includes several texts and their speech unit labels. The number of texts in the first training sample is greater than the number of texts in the second training sample. The first neural network model learns language expression information through a self-supervised pre-training process. A pre-defined second neural network model can be trained using knowledge distillation based on the first training sample and the first neural network model to obtain a language model that has learned language expression information from the first neural network model. The size of the language model is smaller than the size of the first neural network model. Then, the language model is trained in a supervised manner using the second training sample to obtain the speech unit prediction model. This embodiment takes into account that in related technologies, there may be a first neural network model that has learned language expression information but may not be suitable for the application scenario of the speech unit prediction model. Therefore, knowledge distillation can be used to transfer language expression information from a large-scale first neural network model to a small-scale language model to adapt to the application scenario of the speech unit prediction model. On this basis, only an appropriate number of second training samples are needed to perform supervised training on the language model, allowing the language model that has learned language expression information to further learn the mapping relationship between text and speech units, and finally obtain the speech unit prediction model. Even when the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, which also helps to reduce the annotation cost.

[0070] Please see Figure 1 , Figure 1 An exemplary structural diagram of an interactive system is provided. The interactive system includes a server 10 and a client 20. The server 10 and client 20 can be connected via a network, such as the client 20 connecting to the network via GPRS, 4G, 5G, or Wi-Fi.

[0071] The server 10 can be deployed on a cloud server or on a server dedicated to building, distributing and managing speech synthesis.

[0072] In different interaction scenarios, the electronic devices on which the client 20 is located also vary. These electronic devices include, but are not limited to: voice navigation devices, story machines, smart speakers, dictionary pens, wearable devices (e.g., watches, glasses, gloves, headwear (e.g., hats, helmets, virtual reality headsets, augmented reality headsets, head-mounted devices (HMDs), headbands), pendants, armbands, leg rings, shoes, vests), and also mobile communication devices, namely: mobile phones or smartphones, as well as personal computers, PADs, iPads and other terminal devices.

[0073] The training method for the speech unit prediction model provided in this embodiment can be executed by a server. The trained speech unit prediction model is used to process the input text to obtain a speech unit sequence, which can be applied in speech synthesis scenarios. Specifically, if the speech synthesis process is performed on the server, the trained speech unit prediction model can be deployed on the server; if the speech synthesis process is performed on the client, the trained speech unit prediction model can be deployed on the client. Of course, in addition to deploying the speech unit prediction model, the speech synthesis process also requires the deployment of an acoustic model and a vocoder. The acoustic model is used to convert the speech unit sequence into acoustic features, and the vocoder is used to convert the acoustic features into speech data.

[0074] The training method for the speech unit prediction model will be explained next:

[0075] Please see Figure 2 as well as Figure 3 , Figure 2 This is a flowchart illustrating a training method for a speech unit prediction model provided in an embodiment of this specification. The method can be executed by a server and includes:

[0076] In S201, a first training sample and a second training sample are obtained; wherein, the first training sample includes several texts, and the second training sample includes several texts and their speech units; the number of texts in the first training sample is greater than the number of texts in the second training sample.

[0077] In S202, the first training sample is used to perform self-supervised pre-training on the preset neural network model to obtain a language model that has learned language expression information from the first training sample.

[0078] In S203, the language model is trained in a supervised manner using the second training sample to obtain a speech unit prediction model; wherein, the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0079] In this embodiment, a self-supervised learning approach can be used to pre-train the neural network model. The self-supervised learning process does not require manual annotation, and a large amount of text can be obtained for pre-training, allowing the neural network model to fully learn the language expression information and obtain a language model. On this basis, only an appropriate number of second training samples are needed to perform supervised training on the language model, allowing the language model that has learned the language expression information to further learn the mapping relationship between text and speech units, and finally obtain a speech unit prediction model. Even when the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, while also helping to reduce annotation costs.

[0080] For example, the texts in the first training sample include texts obtained from the Internet using web crawling technology and texts obtained from the developer's internal operating data; wherein, the texts obtained from the Internet are far more numerous than the texts obtained from the developer's internal operating data. Combining these two parts of data (which can reach hundreds of millions) for self-supervised learning enables the language model to fully learn language expression information from massive amounts of text. The language expression information includes, but is not limited to, semantic information, syntactic information, and contextual relationships.

[0081] For example, some of the texts in the second training sample include texts obtained from the developer's internal operating data (which may involve tens of thousands of texts). Only a suitable amount of text needs to be labeled, which helps to reduce labeling costs.

[0082] In some embodiments, the pre-training process is illustrated below: The server can replace the text at random positions in the first training sample with mask symbols to obtain masked text. Then, using the masked text as input and the text before masking as labels, the following process is executed cyclically until the loop termination condition is met: using the current neural network model to predict the text at the masked positions in the masked text to obtain predicted text; if the loop termination condition is not met, adjusting the parameters of the current neural network model based on the predicted text and the text before masking to obtain an adjusted neural network model; wherein, the adjusted neural network model serves as the current neural network model for the next loop process.

[0083] For example, the loss function involved in the training process may include a mask reconstruction loss function, which aims to enable the neural network model to predict the masked input based on contextual information. When calculating the mask reconstruction loss function, the reconstruction error can be calculated based on the difference between the predicted text and the text before the mask, for example, using a cross-entropy loss function or a mean squared error loss function.

[0084] Besides the mask reconstruction loss function, other self-supervised objective functions can be combined to further improve pre-training performance, such as the context continuity loss function. Different tasks and datasets may require different loss function choices, and the specific selection should be adjusted according to the actual situation and pre-training objectives.

[0085] In self-supervised pre-training, the input text is typically segmented into a series of segments or windows, each containing a certain number of characters. The continuity loss function leverages the relationships between these segments to provide additional supervision. Specifically, for each input segment, the model predicts the next word or character based on the context. Then, the continuity loss is calculated using the difference between the prediction and the actual next word or character. This loss encourages the model to accurately predict words or characters in the context and to capture local continuity features in the input text. The continuity loss can be calculated using the cross-entropy loss function.

[0086] The loop termination conditions include, but are not limited to, the number of loops reaching a preset number, or the difference between the predicted text and the text in front of the mask being less than a preset threshold.

[0087] If the loop termination condition is met, the current neural network model in the last loop is used as the language model. In this embodiment, during pre-training, the neural network model learns the representation of the input text by learning contextual relationships. By predicting words or characters at mask positions, the neural network model gradually learns to understand semantic and syntactic information and can generate reasonable representations.

[0088] For example, the language model includes, but is not limited to, at least one of the Transformer model, BERT (Bidirectional Encoder Representations from Transformers) model, and GPT (Generative Pre-trained Transformer) model. The Transformer model is a neural network model based on a self-attention mechanism, which can be used for natural language processing and other sequence-to-sequence tasks. The BERT model is a pre-trained natural language processing model. The GPT model is a pre-trained generative language model.

[0089] In some embodiments, after obtaining the language model, the server can use the second training samples to perform supervised training on the language model to obtain a speech unit prediction model. The speech unit prediction model is used to process the input text to obtain a speech unit sequence. For example, if the input text is "Hello, how's the weather today?", the speech unit prediction model outputs the speech unit sequence "nih ao j in ti an ti an qiz en m ey ang".

[0090] Supervised learning is a machine learning task that infers functions from labeled training datasets. It uses samples with known characteristics as training sets to build a mathematical model (such as a discriminant model in pattern recognition, a weight model in artificial neural networks, etc.), and then uses the established model to predict unknown samples.

[0091] This section provides an illustrative example of a supervised training process: Using text as input and the corresponding speech units as labels, the following process is repeated until a loop termination condition is met: The current language model is used to predict the predicted speech units corresponding to the text; if the loop termination condition is not met, the parameters of the current language model are adjusted based on the predicted speech units and their labels to obtain an adjusted language model; wherein, the adjusted language model serves as the current language model for the next iteration. During training, the server can calculate a loss function based on the predicted speech units and their labels, and then adjust the parameters of the current language model in reverse based on the calculated loss value. Loss functions include, but are not limited to, mean squared error loss, cross-entropy loss, or mean absolute error loss, etc., and this embodiment does not impose any limitations on these.

[0092] The loop termination conditions in the supervised training process include, but are not limited to, the number of loops reaching a preset number, or the difference between the predicted speech unit and the speech unit label being less than a preset threshold. When the loop termination conditions are met, the server can use the current language model from the last loop as the speech unit prediction model. This embodiment enables the language model, having learned linguistic expression information, to further learn the mapping relationship between text and speech units, ultimately obtaining the speech unit prediction model. Since the language model has already pre-learned linguistic expression information, the prediction accuracy of the speech unit prediction model can be improved even with a small number of second training samples, while also reducing annotation costs.

[0093] In some embodiments, please refer to Figure 1 The trained speech unit prediction model will be deployed on the server or client side depending on different speech synthesis needs, and the computing resources of different electronic devices are also different.

[0094] If the trained speech unit prediction model is deployed on a server, the size of the language model can be pre-determined based on the server's computing resources before pre-training to adapt to the server's computing resources. The size of the language model is positively correlated with the computing resources of the client or the server. The more computing resources available, the larger the size of the language model can be set to improve the accuracy of the model output. Conversely, the fewer computing resources available, the smaller the size of the language model can be set to improve the speed of the model output. This ensures that the final trained speech unit prediction model meets the server's requirements.

[0095] If the trained speech unit prediction model is deployed on the client, in order to adapt to the client's computing resources, the size of the language model can be pre-determined based on the client's computing resources before pre-training. The size of the language model is positively correlated with the computing resources of the client or the server, so that the finally trained speech unit prediction model meets the client's needs.

[0096] The size of the language model reflects information such as the number of parameters and layers. The larger the number of parameters, the larger the size of the language model, and / or the more layers, the larger the size of the language model.

[0097] Please see Figure 4 as well as Figure 5 , Figure 4 This is a flowchart illustrating a training method for a speech unit prediction model provided in an embodiment of this specification. The method can be executed by a server and includes:

[0098] In S401, a first training sample, a second training sample, and a first neural network model are acquired; wherein, the first training sample includes several texts, the second training sample includes several texts and their speech units, and the number of texts in the first training sample is greater than the number of texts in the second training sample; the first neural network model learns language expression information through a self-supervised pre-training process.

[0099] In S402, a pre-defined second neural network model is trained by knowledge distillation based on the first training sample and the first neural network model to obtain a language model that has learned language expression information from the first neural network model; the size of the language model is smaller than the size of the first neural network model.

[0100] In S403, the language model is trained in a supervised manner using the second training sample to obtain a speech unit prediction model; wherein, the speech unit prediction model is used to process the input text to obtain a speech unit sequence.

[0101] This embodiment considers that in related technologies, there may be a first neural network model that has learned language expression information, but due to its large scale, it may not be suitable for the application scenario of the speech unit prediction model. Therefore, the language expression information can be transferred from the large-scale first neural network model to a small-scale language model through the knowledge distillation process to adapt it to the application scenario of the speech unit prediction model. On this basis, only an appropriate number of second training samples are needed to perform supervised training on the language model, allowing the language model that has learned language expression information to further learn the mapping relationship between text and speech units, and finally obtain the speech unit prediction model. Even when the second training samples are small, the prediction accuracy of the speech unit prediction model can be improved by the above method, which also helps to reduce the annotation cost.

[0102] For example, the text in the first training sample includes text obtained from the internet using web crawling technology and text obtained from the developer's internal operational data; wherein, the text obtained from the internet is significantly more extensive than the text obtained from the developer's internal operational data. The first training sample contains at least a portion of the text used to train the first neural network model; for example, text used to train the first neural network model can be obtained from the internet using web crawling technology, so that the language model can learn the knowledge of the first neural network model. Simultaneously, the first training sample includes text obtained from the developer's internal operational data, so that the language model can also learn language expressions that meet the developer's business needs.

[0103] For example, some texts in the second training sample include texts obtained from the developer's internal operating data (which may involve tens of thousands of texts) for use in the subsequent supervised training process. Only a suitable amount of text needs to be labeled, which helps to reduce labeling costs.

[0104] The first neural network model can be a pre-trained model obtained from the network or other means. For example, the first neural network model can be a model that has learned language expression information through a self-supervised pre-training process. The specific pre-training process can be found in the description above and will not be repeated here. To improve the accuracy of the model output, the first neural network model may be a relatively large model, such as having a large number of parameters and / or layers, which may not be suitable for the application scenarios of speech unit prediction models. For example, when the speech unit prediction model is deployed on the client, the client's computing resources are limited and cannot run a large-scale speech unit prediction model. If the first neural network model is directly used for supervised training, the resulting speech unit prediction model will also be relatively large, obviously unsuitable for deployment on the client. Similarly, when the speech unit prediction model is deployed on the server, the computing resources of the server running the speech unit prediction model may differ from those of the computing device running the first neural network model, making it impossible to directly reuse the first neural network model.

[0105] Therefore, the embodiments of this specification implement knowledge distillation training of a preset second neural network model based on the first training samples and the first neural network model to obtain a language model that learns language expression information from the first neural network model; the size of the language model is smaller than the size of the first neural network model, and thus the second training samples can be used to perform supervised training on the language model to obtain a speech unit prediction model that meets the requirements.

[0106] In some embodiments, please refer to Figure 1 The trained speech unit prediction model will be deployed on the server or client side depending on different speech synthesis needs, and the computing resources of different electronic devices are also different.

[0107] If the trained speech unit prediction model is deployed on the server, in order to adapt to the server's computing resources, the size of the language model can be pre-determined based on the server's computing resources before knowledge distillation training. The size of the language model (or the second neural network model) is positively correlated with the computing resources of the client or the server. The more computing resources available, the larger the size of the language model can be set to improve the accuracy of the model output. Conversely, the fewer computing resources available, the smaller the size of the language model can be set to improve the speed of the model output. This ensures that the final trained speech unit prediction model meets the server's requirements.

[0108] If the trained speech unit prediction model is deployed on the client, in order to adapt to the client's computing resources, the size of the language model can be pre-determined based on the client's computing resources before knowledge distillation training. The size of the language model (or the second neural network model) is positively correlated with the computing resources of the client or the server, so that the finally trained speech unit prediction model meets the client's needs.

[0109] The size of the language model reflects information such as the number of parameters and layers. The larger the number of parameters, the larger the size of the language model, and / or the more layers, the larger the size of the language model.

[0110] After determining the size of the language model (or the second neural network model), knowledge distillation can be performed. Knowledge distillation improves the performance of a smaller model by transferring knowledge from a complex, large model to a simplified one. This process is achieved by using the output of the large model as an additional training objective, which is then used along with the original labels to train the smaller model. Through knowledge distillation, the smaller model can acquire richer knowledge and representational capabilities from the larger model, achieving performance close to or even exceeding that of the larger model while maintaining relatively low complexity. This is particularly useful for deploying models in resource-constrained environments or improving the efficiency of models on edge devices.

[0111] Here is an illustrative explanation of the process of obtaining a language model: The server can replace the text at random positions in the first training sample with mask symbols to obtain the masked text. Then, the following steps are executed repeatedly until the loop termination condition is met: using the current second neural network model to predict the text at the mask position in the masked text to obtain the first predicted text; and using the first neural network model to predict the text at the mask position in the masked text to obtain the second predicted text; if the loop termination condition is not met, the parameters of the current second neural network model are adjusted according to the first predicted text, the second predicted text, and the text before masking to obtain the adjusted second neural network model; wherein, the adjusted second neural network model serves as the current second neural network model for the next loop process.

[0112] During the iteration, a distillation loss function is used to measure the difference between the predictions of the second neural network model and the output of the first neural network model. The distillation loss function can be calculated based on the output probability distributions of the first and second neural network models. As training progresses, optimization of the distillation loss function will gradually bring the second neural network model closer to the prediction results of the first neural network model. The distillation loss function is used to assess the difference between the predictions of the second neural network model and the output of the first neural network model. It also needs to consider the difference between the predictions of the second neural network model and the original text labels. Therefore, the loss can be calculated based on the first and second predicted texts, as well as based on the first predicted text and the text before the mask. The two losses can then be weighted and combined, with the weights determined according to actual requirements.

[0113] The loop termination conditions include, but are not limited to, the number of loops reaching a preset number, or the difference between the predicted text and the text in front of the mask being less than a preset threshold.

[0114] If the loop termination condition is met, the current second neural network model in the last loop can be used as the language model. In this embodiment, through knowledge distillation, the second neural network model can obtain richer knowledge and representation capabilities from the first neural network model, and while maintaining relatively low complexity, it has the ability to approach or even exceed the performance of the first neural network model. Since knowledge transfer is achieved based on the pre-trained first neural network model, relative to... Figure 2 The pre-training process described in this embodiment, which starts from scratch, allows for a faster acquisition of a language model.

[0115] For example, the language model includes, but is not limited to, at least one of the Transformer model, BERT (Bidirectional Encoder Representations from Transformers) model, and GPT (Generative Pre-trained Transformer) model. The Transformer model is a neural network model based on a self-attention mechanism, which can be used for natural language processing and other sequence-to-sequence tasks. The BERT model is a pre-trained natural language processing model. The GPT model is a pre-trained generative language model.

[0116] In some embodiments, after obtaining the language model, the server can use the second training samples to perform supervised training on the language model to obtain a speech unit prediction model. The speech unit prediction model is used to process the input text to obtain a sequence of speech units.

[0117] This section provides an illustrative example of a supervised training process: Using text as input and the corresponding speech units as labels, the following process is repeated until a loop termination condition is met: The current language model is used to predict the predicted speech units corresponding to the text; if the loop termination condition is not met, the parameters of the current language model are adjusted based on the predicted speech units and their labels to obtain an adjusted language model; wherein, the adjusted language model serves as the current language model for the next iteration. During training, the server can calculate a loss function based on the predicted speech units and their labels, and then adjust the parameters of the current language model in reverse based on the calculated loss value. Loss functions include, but are not limited to, mean squared error loss, cross-entropy loss, or mean absolute error loss, etc., and this embodiment does not impose any limitations on these.

[0118] The loop termination conditions in the supervised training process include, but are not limited to, the number of loops reaching a preset number, or the difference between the predicted speech unit and the speech unit label being less than a preset threshold. When the loop termination conditions are met, the server can use the current language model from the last loop as the speech unit prediction model. This embodiment enables the language model, having learned linguistic expression information, to further learn the mapping relationship between text and speech units, ultimately obtaining the speech unit prediction model. Since the language model has already pre-learned linguistic expression information, the prediction accuracy of the speech unit prediction model can be improved even with a small number of second training samples, while also reducing annotation costs.

[0119] In some embodiments, the application of the speech unit prediction model is illustrated herein by way of example: Please refer to Figure 6 This specification provides a speech synthesis method, which can be executed by a server or a client. The method includes:

[0120] In S601, the text to be processed is obtained.

[0121] In S602, the text to be processed is converted into a sequence of speech units using a speech unit prediction model; wherein the speech unit prediction model is trained based on any of the training methods described above.

[0122] In S603, the speech unit sequence is processed using an acoustic model to obtain acoustic features.

[0123] In S604, the acoustic features are converted into speech data using a vocoder.

[0124] This embodiment applies the speech unit prediction model trained above during the speech synthesis process. This speech unit prediction model can convert text into a speech unit sequence in one step, without going through multiple steps such as word segmentation, character-to-phonetic conversion, text regularization, prosody prediction, and polyphonic character prediction. This simplifies the process of converting text into a speech unit sequence, which helps reduce error accumulation and improve the success rate of speech synthesis. At the same time, the one-step conversion process also helps improve the efficiency of speech synthesis.

[0125] In some embodiments, if the speech synthesis method is executed by a server, the speech unit prediction model, the acoustic model, and the vocoder are deployed on the server. After acquiring the speech data, the server can send it to the client so that the speech data can be played on the client.

[0126] In other embodiments, if the speech synthesis method is executed by a client, the speech unit prediction model, the acoustic model, and the vocoder are deployed on the client, and the client can directly play the speech data after acquiring it.

[0127] In some embodiments, the training process of the acoustic model and the vocoder is illustrated by way of example: First, a speaker corpus is constructed, which consists of speech and its corresponding text. The speech is converted into acoustic features and the text is converted into a sequence of speech units. Through supervised learning, the acoustic model learns the mapping relationship from the speech unit sequence to the acoustic features, and the vocoder learns the mapping relationship from the acoustic features to the speech.

[0128] For example, considering that the speech synthesis method of this specification can be applied to different scenarios, such as news playback, storytelling, and voice navigation, and considering that different scenarios have different language expressions, multiple speaker corpora can be constructed for supervised training during the training phase, including but not limited to news speaker corpora, navigation speaker corpora, and storytelling speaker corpora, so that the trained acoustic model and the acoustic model can be applied to different scenarios.

[0129] In some embodiments, please refer to Figure 7 The acoustic model includes a decoder, a parameter adjuster, and an encoder; the speech unit sequence is a discrete character sequence, and the decoder is used to process the speech unit sequence to obtain a more compact representation vector representing contextual semantic information; the parameter adjuster is used to process the representation vector to obtain speech information, which includes at least one of the following: speech duration, volume, and pitch; the decoder is used to process the representation vector and the speech information to obtain acoustic features.

[0130] For example, the encoder is composed of multiple layers of Transformer components stacked together; the parameter adjuster includes a convolutional neural network; the encoder is composed of multiple layers of Conformer structures stacked together.

[0131] In some embodiments, the vocoder can be a user-specific vocoder or a vocoder shared by multiple users. The vocoder can be an LPCNet vocoder, a WaveRNN vocoder, a HiFi-gan vocoder, etc., without any limitations.

[0132] In some embodiments, when the speech synthesis method is applied to a client, the server can deploy the speech unit prediction model, the acoustic model, and the vocoder to the client. To adapt to the client's computing resources, before deploying the speech unit prediction model, the acoustic model, and the vocoder to the client, the server further performs compression processing on at least one of the speech unit prediction model, the acoustic model, and the vocoder according to the client's computing resources.

[0133] For example, the compression process includes at least one of pruning and quantization; the pruning process indicates the removal of redundant parameters or redundant networks in the neural network model; the quantization process indicates the reduction of the number of bits in the parameters and / or activation values ​​in the neural network model.

[0134] Pruning includes parametric pruning and network pruning.

[0135] Parameter pruning refers to reducing the number of parameters in a model by removing redundant or unimportant parameters. Common pruning methods include structured pruning and unstructured pruning. Taking parameter pruning of a speech unit prediction model as an example, firstly, the importance of each parameter in the speech unit prediction model can be evaluated to determine its significance. Common parameter importance evaluation methods include structured pruning methods (such as L1 regularization) and unstructured pruning methods (such as sensitivity analysis and Taylor approximation). Then, based on the parameter importance evaluation results, a certain proportion of parameters are selected for pruning; pruning can be done by directly setting parameters to zero (sparse pruning) or deleting unimportant parameters; the pruned model will become sparser, i.e., it will contain more zero parameters. Finally, the pruned model can be fine-tuned, i.e., retrained using the second training sample mentioned above, to restore or improve its performance. The purpose of fine-tuning is to minimize performance loss while maintaining the sparsity of the model.

[0136] Network pruning is similar to parametric pruning, but it involves more than just pruning weights; it also includes pruning neurons, pruning layers, and so on. By pruning the entire network structure, the size and computational cost of the model can be reduced.

[0137] Quantization refers to converting the parameters and activation values ​​in a neural network model from floating-point numbers to fixed-point numbers or lower-precision floating-point numbers. Taking the quantization of parameters in a speech unit prediction model as an example, the floating-point parameters of the speech unit prediction model can be mapped to fixed-digit integers or fixed-point decimals, and then scaled back to floating-point numbers using a scaling factor. After quantization, the quantized model can be fine-tuned, i.e., retrained using the second training sample mentioned above, to restore or improve its performance. The purpose of fine-tuning is to minimize performance loss while maintaining the quantization of the model.

[0138] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0139] In some embodiments, please refer to Figure 1 This specification also provides a server-side implementation that can execute any of the methods described above.

[0140] For example, the speech unit prediction model can be trained on the server, or the speech unit prediction model, acoustic model, and vocoder can be deployed on the server so that the server can perform the speech synthesis process.

[0141] Of course, the training process of the speech unit prediction model can also be performed by other computing devices that are different from the server, and this embodiment does not impose any restrictions on this.

[0142] In some embodiments, please refer to Figure 1 This specification also provides a client that can execute the above-described speech synthesis method.

[0143] For example, a speech unit prediction model, an acoustic model, and a vocoder can be deployed on the client so that the client can perform the speech synthesis process.

[0144] In some embodiments, this specification also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor implements the method described in any one of the above embodiments by executing the executable instructions.

[0145] For example, Figure 8 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. Please refer to... Figure 8 At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, memory 808, and non-volatile memory 810, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into memory 808 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0146] In some embodiments, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the method as described in any of the preceding embodiments.

[0147] In some embodiments, this specification also provides a computer program product that, when executed by a processor, implements the steps of the method as described in any of the foregoing embodiments.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0149] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0150] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0155] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0156] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0157] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A training method for a speech unit prediction model, characterized in that, include: Obtain a first training sample and a second training sample; wherein, the first training sample includes several texts, and the second training sample includes several texts and their speech units; the number of texts in the first training sample is greater than the number of texts in the second training sample; The first training sample is used to perform self-supervised pre-training on a preset neural network model to obtain a language model that has learned language expression information from the first training sample. The language model is trained in a supervised manner using the second training sample to obtain a speech unit prediction model; wherein, the speech unit prediction model is used to process the input text to obtain a speech unit sequence; Wherein, the step of performing self-supervised pre-training on a preset neural network model using the first training samples includes: The text in the first training sample at random positions is replaced with mask symbols to obtain the masked text. Using the masked text as input and the unmasked text as labels, the following process is repeated until the loop termination condition is met: using the current neural network model to predict the text at the masked position in the masked text, obtaining the predicted text; if the loop termination condition is not met, adjusting the parameters of the current neural network model based on the predicted text and the unmasked text, obtaining the adjusted neural network model; wherein, the adjusted neural network model serves as the current neural network model for the next loop process; If the loop termination condition is met, the current neural network model in the last loop process is used as the language model.

2. A training method for a speech unit prediction model, characterized in that, include: A first training sample, a second training sample, and a first neural network model are obtained; wherein, the first training sample includes several texts, the second training sample includes several texts and their speech units, the number of texts in the first training sample is greater than the number of texts in the second training sample; the first neural network model learns language expression information through a self-supervised pre-training process; Based on the first training samples and the first neural network model, a pre-defined second neural network model is trained by knowledge distillation to obtain a language model that learns language expression information from the first neural network model; the size of the language model is smaller than the size of the first neural network model. The language model is trained in a supervised manner using the second training sample to obtain a speech unit prediction model; wherein, the speech unit prediction model is used to process the input text to obtain a speech unit sequence; Wherein, the first training sample contains at least a portion of text used to train the first neural network model; the step of performing knowledge distillation training on a preset second neural network model based on the first training sample pair and the first neural network model to obtain a language model that has learned linguistic expression information from the neural network model includes: The text in the first training sample at random positions is replaced with mask symbols to obtain the masked text. The following steps are executed repeatedly until the loop termination condition is met: using the current second neural network model to predict the text at the mask position in the text behind the mask, obtaining the first predicted text; and using the first neural network model to predict the text at the mask position in the text behind the mask, obtaining the second predicted text; if the loop termination condition is not met, the parameters of the current second neural network model are adjusted according to the first predicted text, the second predicted text, and the text before the mask, to obtain the adjusted second neural network model; wherein, the adjusted second neural network model serves as the current second neural network model for the next loop process; If the loop termination condition is met, the current second neural network model in the last loop process is used as the language model.

3. The method according to any one of claims 1 to 2, characterized in that, The speech unit prediction model is used for deployment on a client or server; The size of the language model is determined based on the computing resources of the client or the server, and the size of the language model is positively correlated with the computing resources of the client or the server.

4. The method according to any one of claims 1 to 2, characterized in that, The language model includes at least one of the Transformer model, BERT model, and GPT model; and / or The first training sample includes text obtained from the web using web crawling technology and text obtained from the developer's internal operating data; the second training sample includes text obtained from the developer's internal operating data.

5. A speech synthesis method, characterized in that, include: Get the text to be processed; The text to be processed is converted into a sequence of speech units using a speech unit prediction model; wherein the speech unit prediction model is trained based on the training method described in any one of claims 1 to 4. The speech unit sequence is processed using an acoustic model to obtain acoustic features; The acoustic features are converted into speech data using a vocoder.

6. The method according to claim 5, characterized in that, The acoustic model includes a decoder, a parameter adjuster, and an encoder; The decoder is used to process the speech unit sequence to obtain a representation vector that represents contextual semantic information; The parameter adjuster is used to process the representation vector to obtain speech information, which includes at least one of the following: speech duration, volume, and pitch. The decoder is used to process the representation vector and the speech information to obtain acoustic features.

7. The method according to claim 5 or 6, characterized in that, The method is applied to the client, and the speech unit prediction model, the acoustic model, and the vocoder are deployed from the server to the client. Before deploying the speech unit prediction model, the acoustic model, and the vocoder to the client, the server is further configured to compress at least one of the speech unit prediction model, the acoustic model, and the vocoder according to the client's computing resources. The compression process includes at least one of pruning and quantization; the pruning process indicates the removal of redundant parameters or redundant networks in the neural network model; the quantization process indicates the reduction of the number of bits in the parameters and / or activation values ​​in the neural network model.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 7 by executing the executable instructions.

Images