Method for training a speech synthesis model, speech synthesis method, apparatus, and electronic device

Abstract

This invention provides a method for training a speech synthesis model, a speech synthesis method, and apparatus and electronic devices related thereto. [Solution] The training method includes the step of acquiring training data. The training samples in the training data include style description sample text, input sample text, output sample audio, and style dimension data of the output sample audio. The method also includes the step of acquiring an initial speech synthesis model, and the step of performing a training process on the speech synthesis model based on the style description sample text, input sample text, output sample audio, and style dimension data to acquire a trained speech synthesis model. [Effect] By combining style dimension data with the training process for the speech synthesis model, more features can be considered during the training of the speech synthesis model, thereby improving the accuracy of the trained speech synthesis model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to technical fields such as deep learning, natural language processing, voice technology, large-scale models, etc., and particularly to a method for training a voice synthesis model, a voice synthesis method, an apparatus, and an electronic device.

Background Art

[0002] The current voice synthesis model is mainly obtained by training an initial voice synthesis model by combining a style description sample text, an input sample text, and an output sample voice. Here, the loss function of the voice synthesis model is determined based on the output sample voice and the predicted sample voice. Since few features are considered, the accuracy of the trained voice synthesis model is low.

Summary of the Invention

[0003] This disclosure provides a method for training a voice synthesis model, a voice synthesis method, an apparatus, and an electronic device.

[0004] According to an aspect of this disclosure, a method for training a voice synthesis model is provided. The method includes: obtaining training data, where the training samples in the training data include a style description sample text, an input sample text, an output sample voice, and style dimension data of the output sample voice; obtaining an initial voice synthesis model; and performing a training process on the voice synthesis model based on the style description sample text, the input sample text, the output sample voice, and the style dimension data to obtain a trained voice synthesis model.

[0005] Another aspect of the present disclosure provides a speech synthesis method comprising: acquiring input text and style description text; inputting the style description text into a style control network in a speech synthesis model to acquire a style vector output by the style control network, wherein the speech synthesis model is obtained by training on style description sample text, input sample text, output sample speech and style dimension data of the output sample speech; and inputting the style vector and the input text into a backbone network in the speech synthesis model to acquire output speech corresponding to the input text, which is output by the backbone network.

[0006] In another aspect of the present disclosure, a speech synthesis model training apparatus is provided, the apparatus comprising: a first acquisition module for acquiring training data, wherein the training samples in the training data include style description sample text, input sample text, output sample audio, and style dimension data of the output sample audio; a second acquisition module for acquiring an initial speech synthesis model; and a training processing module for performing a training process on the speech synthesis model based on the style description sample text, input sample text, output sample audio, and style dimension data to acquire a trained speech synthesis model.

[0007] In another aspect of the present disclosure, a speech synthesis device is provided, the device comprising: a first acquisition module for acquiring input text and style description text; a second acquisition module for inputting the style description text into a style control network in a speech synthesis model and acquiring a style vector output by the style control network, wherein the speech synthesis model is obtained by training on style description sample text, input sample text, output sample speech and style dimension data of the output sample speech; and a third acquisition module for inputting the style vector and the input text into a backbone network in the speech synthesis model and acquiring an output speech corresponding to the input text output by the backbone network.

[0008] In another aspect of the present disclosure, an electronic device is provided, comprising at least one processor and a memory communicably connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the execution of the instructions by the at least one processor causes the at least one processor to execute the above-mentioned method for training the speech synthesis model of the present disclosure or the above-mentioned method for speech synthesis of the present disclosure.

[0009] According to another aspect of the present disclosure, a non-temporary computer-readable storage medium is provided which stores computer instructions, such computer instructions cause a computer to perform the above-mentioned method for training the speech synthesis model or the above-mentioned speech synthesis method of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program is provided and the computer program is executed by a processor, thereby realizing the steps of the above-described method for training a speech synthesis model or the steps of the above-described method for speech synthesis.

[0011] The content described in this section is not intended to identify any essential or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure are better understood from the following specification. [Brief explanation of the drawing]

[0012] The drawings are provided to better understand this solution and do not limit the scope of this disclosure. [Figure 1] This is a schematic diagram of the first embodiment of the present disclosure. [Figure 2] This is a schematic diagram of a second embodiment of the present disclosure. [Figure 3] This is a schematic diagram of a third embodiment of the present disclosure. [Figure 4] This is a schematic diagram of the training process for a style control network. [Figure 5] This is a schematic diagram of the backbone network training. [Figure 6] This is a schematic diagram of a fourth embodiment of the present disclosure. [Figure 7] This is a schematic diagram of the fifth embodiment of the present disclosure. [Figure 8] This is a block diagram of an electronic device for realizing a speech synthesis model training method or speech synthesis method according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0013] The following description, in conjunction with the drawings, includes various details of the embodiments of this disclosure to facilitate understanding; however, these should be considered merely illustrative. Therefore, those skilled in the art should be aware that various changes and modifications can be made to the embodiments described, without departing from the scope and spirit of this application. Similarly, for clarity and conciseness, well-known functions and structures are omitted in the following description.

[0014] Current speech synthesis models are primarily obtained by training an initial speech synthesis model by combining style description sample text, input sample text, and output sample speech. Here, the loss function of the speech synthesis model is determined based on the output sample speech and predicted sample speech, and because few features are considered, the accuracy of the trained speech synthesis model is low.

[0015] To address the above issues, this disclosure proposes a method for training a speech synthesis model, a speech synthesis method, an apparatus, and an electronic device.

[0016] Figure 1 is a schematic diagram of a first embodiment of the present disclosure. The speech synthesis model training method of the embodiment of the present disclosure is applicable to a speech synthesis model training device, which can be configured to enable the electronic device to perform the speech synthesis model training function.

[0017] Here, the electronic device may be any device equipped with computing capabilities, such as a personal computer (PC), a mobile terminal, or a server. The mobile terminal may be a hardware device equipped with various operating systems, touchscreens, and / or displays, such as an in-car device, a mobile phone, a tablet, a personal digital assistant, a wearable device, a smart speaker, a server, or a server cluster.

[0018] Here, the training device for the speech synthesis model may be software on an electronic device, such as training software for a speech synthesis model. In the following embodiment, we will explain using the example that the execution entity is an electronic device.

[0019] As shown in Figure 1, the training method for the speech synthesis model can include the following steps.

[0020] In step 101, training data is acquired. The training samples in the training data include style description sample text, input sample text, output sample audio, and style dimension data of the output sample audio.

[0021] In an embodiment of the present disclosure, the style dimension data of the output sample audio may include a sample style sequence in at least one style dimension and / or sample style content in at least one style dimension.

[0022] Here, in one example, the style dimension data of the output sample audio may include a sample style sequence in at least one style dimension. In another example, the style dimension data of the output sample audio may include sample style content in at least one style dimension. In another example, the style dimension data of the output sample audio may include a sample style sequence and sample style content in at least one style dimension.

[0023] Here, when the style dimension data of the output sample audio includes sample style content in at least one style dimension, the sample style content in each style dimension can be determined by at least one of manual labeling or using a style recognition model of the corresponding style dimension.

[0024] Here, if the style dimension data of the output sample audio includes a sample style sequence and sample style content in at least one style dimension, for example, the process by which the electronic device performs step 101 may involve, for example, obtaining a style description sample text, input sample text and output sample audio, performing a style extraction process in at least one style dimension for each audio frame in the output sample audio to obtain a sample style sequence in at least one style dimension, and determining the sample style content in at least one style dimension based on the sample style sequence in at least one style dimension.

[0025] In embodiments of the present disclosure, the style dimension includes at least one of pitch, volume, speech rate, pitch variance, and emotion type.

[0026] Here, using pitch as an example, the method for obtaining the sample style sequence and sample style content in the pitch dimension of the output sample audio may be, for example, by performing a fundamental frequency extraction process on the output sample audio to obtain a fundamental frequency signal sequence containing the fundamental frequency in each audio frame of the output sample audio, performing quantile discretization on the fundamental frequency signal sequence to obtain a pitch sequence in the pitch dimension, which is the sample style sequence in the pitch dimension, and determining the pitch value, which is the sample style content in the pitch dimension, based on the pitch sequence in the pitch dimension. Alternatively, the pitch variance value in the pitch variance dimension can also be determined based on the pitch sequence in the pitch dimension.

[0027] Here, each pitch number in the pitch sequence may include, for example, at least one of the following: very low, low, medium, high, and very high.

[0028] Here, using volume as an example, the method for obtaining the sample style sequence and sample style content in the volume dimension of the output sample audio may be, for example, by performing an energy calculation process on each audio frame in the output sample audio to obtain the energy in each audio frame, thereby obtaining an energy time series sequence, performing a quantile discretization process on the energy time series sequence to obtain a volume sequence in the volume dimension which is the sample style sequence in the volume dimension, and determining the volume value which is the sample style content in the volume dimension based on the volume sequence in the volume dimension.

[0029] Here, each volume value in the volume sequence may include, for example, at least one of the following: very low, low, medium, high, and very high.

[0030] Here, using speech rate as an example, the method for obtaining the sample style sequence and sample style content in the speech rate dimension of the output sample audio may be, for example, to perform phoneme-level forced alignment on the output sample audio to calculate the duration of each phoneme, construct a rhythm feature sequence, determine the speech rate of each audio frame in the output sample audio based on the rhythm feature sequence, obtain a speech rate time series sequence, perform quantile discretization on the speech rate time series sequence to obtain a speech rate sequence in the speech rate dimension which is the sample style sequence in the speech rate dimension, and determine the speech rate value which is the sample style content in the speech rate dimension based on the speech rate sequence in the speech rate dimension.

[0031] Here, the speech rate value in the speech rate sequence may include, for example, at least one of very low, low, medium, high, and very high.

[0032] Here, using emotion types as an example, the method for obtaining the sample style sequence and sample style content in the emotion type dimension of the output sample audio may be, for example, by performing emotion feature vector extraction processing on each audio frame of the output sample audio to obtain an emotion feature vector sequence, performing spectral clustering processing on the emotion feature vector sequence to obtain an emotion type sequence in the emotion type dimension which is the sample style sequence in the emotion type dimension, and determining the emotion type which is the sample style content in the emotion type dimension based on the emotion type sequence in the emotion type dimension.

[0033] Here, the emotional types can include at least one of the following, for example, joy, anger, sadness, fear, surprise, disgust, sorrow, heaviness, warmth, humor, and rawness, and can be set according to actual needs; they are not particularly limited here.

[0034] Here, by performing style extraction processing in at least one style dimension for each audio frame in the output sample audio, the electronic device can combine the sample style sequence and / or sample style content in at least one style dimension to train the speech synthesis model. This allows the speech synthesis model to be trained in combination with dimensional data in at least one style dimension, resulting in a greater number of features being considered and improving the accuracy of the trained speech synthesis model.

[0035] Here, the flexible setting of multiple style dimensions allows electronic devices to select the necessary style dimensions according to their actual needs and use them for training speech synthesis models, thereby improving the flexibility of speech synthesis model training.

[0036] In step 102, an initial speech synthesis model is obtained.

[0037] In the embodiments of this disclosure, the speech synthesis model may include a backbone network and a style control network, the style control network for performing style extraction on an input style description text to obtain a style feature vector, and the backbone network for performing speech synthesis processing by combining the input text and the style feature vector to obtain output speech.

[0038] Here, by configuring the backbone network and style control network, the electronic device can perform accurate training on the backbone network and style control network in combination with the training data, thereby improving the accuracy of both the backbone network and the style control network, and further improving the accuracy of the trained speech synthesis model.

[0039] In embodiments of the present disclosure, the backbone network may include an encoding network and a decoding network, the encoding network being provided with a hierarchical cross-attention mechanism for determining a predicted style sequence in at least one style dimension based on an intermediate feature vector and a second predicted style vector, the intermediate feature vector being an intermediate vector in the processing steps of the encoding network.

[0040] Here, by setting up a hierarchical cross-attention mechanism, the coding network can extract more features from the second predicted style vector, further improving the accuracy of the trained speech synthesis model.

[0041] Here, the backbone network could be, for example, a fish-speech text-to-speech conversion model. The style control network could be, for example, a BERT model.

[0042] In step 103, the speech synthesis model is trained based on the style description sample text, input sample text, output sample audio, and style dimension data to obtain the trained speech synthesis model.

[0043] In one embodiment of the present disclosure, the process by which the electronic device performs step 103 may be, for example, inputting a style description sample text into a style control network in a speech synthesis model to obtain a predicted style vector output by the style control network, inputting the predicted style vector and the input sample text into a backbone network in a speech synthesis model to obtain predicted style dimension data and output predicted speech output by the backbone network, combining the predicted style dimension data, style dimension data, output predicted speech, and output sample speech to perform parameter tuning on the speech synthesis model to obtain a trained speech synthesis model.

[0044] The method for training a speech synthesis model in the embodiments of this disclosure involves acquiring training data, where the training samples in the training data include style description sample text, input sample text, output sample audio, and style dimension data of the output sample audio, acquiring an initial speech synthesis model, and performing a training process on the speech synthesis model based on the style description sample text, input sample text, output sample audio, and style dimension data to acquire a trained speech synthesis model. Here, by performing the training process on the speech synthesis model in combination with style dimension data, the electronic device can take into account many features during the training of the speech synthesis model, thereby improving the accuracy of the trained speech synthesis model.

[0045] Here, in order to further improve the accuracy of the trained speech synthesis model, the electronic device can progressively train the backbone network and style control network in the speech synthesis model in combination with training samples. First, the style control network is trained, and then the backbone network is trained based on the trained style control network. As shown in Figure 2, Figure 2 is a schematic diagram of a second embodiment of the present disclosure, and the embodiment shown in Figure 2 may include the following steps.

[0046] In step 201, training data is acquired, and the training samples in the training data include style description sample text, input sample text, output sample audio, and style dimension data of the output sample audio, the style dimension data includes style content in at least one style dimension of the output sample audio.

[0047] In step 202, an initial speech synthesis model is obtained.

[0048] In step 203, the style control network in the speech synthesis model is trained based on the style description sample text and the sample style content in at least one style dimension of the output sample speech to obtain the trained style control network.

[0049] In embodiments of this disclosure, the process by which an electronic device performs step 203 may, for example, involve inputting style description sample text into a style control network, obtaining a first predicted style vector output by the style control network, determining predicted style content in at least one style dimension based on the first predicted style vector, and performing parameter tuning on the style control network based on the predicted style content and sample style content in at least one style dimension to obtain a trained style control network.

[0050] Here, the electronic device can input a first predicted style vector into a classification network of at least one style dimension to obtain predicted style content in at least one style dimension.

[0051] Here, the electronic device can determine the value of the loss function based on the predicted style content, sample style content, and the style control network's loss function in at least one style dimension, and then perform parameter tuning on the style control network based on the value of the loss function to obtain a trained style control network.

[0052] Here, by training the style control network in combination with sample style content in at least one style dimension, the style control network can determine accurate style content in at least one style dimension from accurate style vectors derived from style description sample text. This allows the style control network to learn more style-related features and further improve the accuracy of the trained style control network.

[0053] In step 204, the backbone network in the speech synthesis model is trained based on the style description sample text, input sample text, output sample speech, and the trained style control network to obtain the trained backbone network.

[0054] In one embodiment of the present disclosure, the process by which the electronic device performs step 204 may, for example, involve inputting a style description sample text into a trained style control network to obtain a second predicted style vector output by the style control network, inputting the second predicted style vector and the input sample text into a backbone network to obtain an output predicted speech output by the backbone network, and performing parameter tuning on the backbone network based on the output predicted speech and the output sample speech to obtain a trained backbone network.

[0055] Here, the electronic device can specifically determine a loss value based on the predicted output voice and the output sample voice, perform parameter adjustment processing on the backbone network based on the loss value, and obtain a trained backbone network.

[0056] In other examples, the process by which an electronic device performs step 204 may involve, for example, inputting a style description sample text into a trained style control network to obtain a second predicted style vector output by the style control network, inputting the second predicted style vector and the input sample text into a backbone network to obtain a predicted style sequence and output predicted speech in at least one style dimension output by the backbone network, and performing parameter tuning on the backbone network based on the predicted style sequence, sample style sequence, output predicted speech and output sample speech in at least one style dimension to obtain a trained backbone network.

[0057] Here, the electronic device can specifically determine a first loss value based on a predicted style sequence and a sampled style sequence in at least one style dimension, determine a second loss value based on the output predicted speech and the output sampled speech, and perform parameter tuning on the backbone network based on the first and second loss values ​​to obtain a trained backbone network.

[0058] Here, the electronic device combines a predicted style sequence and a sample style sequence in at least one style dimension and performs parameter tuning on the backbone network, enabling the backbone network to learn style-related knowledge, allowing more features to be learned. By performing parameter tuning on the backbone network based on the first and second loss function values, the backbone network can learn style-related knowledge and speech-related knowledge simultaneously, thereby further improving the accuracy of the trained backbone network.

[0059] For further details regarding steps 201 and 202, please refer to steps 101 and 102 in the embodiment shown in Figure 1. A detailed explanation will be omitted here.

[0060] The method for training a speech synthesis model in an embodiment of this disclosure involves acquiring training data, where the training samples in the training data include style description sample text, input sample text, output sample audio, and style dimension data of the output sample audio, the style dimension data includes style content in at least one style dimension of the output sample audio, acquiring an initial speech synthesis model, performing a training process on the style control network in the speech synthesis model based on the style description sample text and the sample style content in at least one style dimension of the output sample audio to acquire a trained style control network, and performing a training process on the backbone network in the speech synthesis model based on the style description sample text, input sample text, output sample audio, and the trained style control network to acquire a trained backbone network. Here, the electronic device can further improve the accuracy of the trained speech synthesis model by training the backbone network and style control network in the speech synthesis model stepwise in combination with the training samples, first training the style control network, and then performing a training process on the backbone network based on the trained style control network.

[0061] Figure 3 is a schematic diagram of a third embodiment of the present disclosure. The speech synthesis method of the embodiment of the present disclosure is applicable to a speech synthesis device, and the device can be configured so that the electronic device can perform a speech synthesis function.

[0062] Here, the electronic device may be any device equipped with computing capabilities, such as a personal computer (PC), a mobile terminal, or a server. The mobile terminal may be a hardware device equipped with various operating systems, touchscreens, and / or displays, such as an in-car device, a mobile phone, a tablet, a personal digital assistant, a wearable device, a smart speaker, a server, or a server cluster.

[0063] Here, the speech synthesis device may be software on an electronic device, such as speech synthesis software. In the following embodiments, the execution entity will be an electronic device as an example.

[0064] As shown in Figure 3, the speech synthesis method may include the following steps.

[0065] In step 301, the input text and style description text are obtained.

[0066] Here, style description text describes the speech style. Examples of style description text include: "express sad emotions," "speak in a solemn tone," "use a lively tone," and "read information in a warm tone."

[0067] In step 302, the style description text is input to the style control network in the speech synthesis model, and the style vector output by the style control network is obtained. The speech synthesis model is then trained based on the style description sample text, input sample text, output sample speech, and style dimension data of the output sample speech.

[0068] In the embodiments of this disclosure, the speech synthesis model may include a backbone network and a style control network, the style control network for performing style extraction on an input style description text to obtain a style feature vector, and the backbone network for performing speech synthesis processing by combining the input text and the style feature vector to obtain output speech.

[0069] Here, the training method for the speech synthesis model can be found in the examples shown in Figures 1 and 2, and will not be explained further here.

[0070] Here, the style dimension data of the output sample audio may include a sample style sequence in at least one style dimension, and / or sample style content in at least one style dimension. Here, the sample style sequence in a style dimension may include sample style content in the style dimension of each audio frame in the output sample audio.

[0071] In step 303, the style vector and input text are input to the backbone network in the speech synthesis model, and the output speech corresponding to the input text, output by the backbone network, is obtained.

[0072] In embodiments of the present disclosure, the backbone network may include an encoding network and a decoding network, the encoding network being provided with a hierarchical cross-attention mechanism for determining a predicted style sequence in at least one style dimension based on an intermediate feature vector and a second predicted style vector, the intermediate feature vector may be an intermediate vector generated during the input text processing process by the encoding network.

[0073] Here, the backbone network may be, for example, a fish-speech text-to-speech conversion model. The style control network may be, for example, a BERT model. In the fish-speech model, the encoding network can consist of a slow autoregressive network and a fast autoregressive network. A hierarchical cross-attention mechanism can be provided for both the slow autoregressive network and the fast autoregressive network.

[0074] Here, slow autoregressive networks are specialized for modeling long-range dependencies in interlingual contexts and are used to extract high-dimensional features. On the other hand, fast autoregressive networks are responsible for the autoregressive generation of low-dimensional acoustic features and are used to extract coarse-grained acoustic features.

[0075] The speech synthesis method of the embodiment of this disclosure acquires input text and style description text, inputs the style description text into a style control network in a speech synthesis model, and obtains a style vector output by the style control network. The speech synthesis model is obtained by training it based on style description sample text, input sample text, output sample speech, and style dimension data of the output sample speech. The style vector and input text are input into a backbone network in the speech synthesis model, and output speech corresponding to the input text is obtained output by the backbone network. Here, since style-related features can be considered in the speech synthesis model, the degree of style matching between the determined output speech and the style description text can be increased, and the accuracy of the output speech can be further improved.

[0076] The following will illustrate with an example. As shown in Figure 4, it is a schematic diagram of the training of a style control network. In Figure 4, the input to BERT (i.e., the style control network) may be a style description text or a text vector corresponding to the style description text. Here, in Figure 4, CLS indicates that BERT will output a Style Embedding (style vector), and T1 to TM represent each character in the style description text. Here, in combination with the Style Embedding, predicted style content in five style dimensions—Emotion, Pitch, Volume, Speaking Rate, and Pitch Variation—can be determined, and sample style content in these five style dimensions can be combined to fine-tune BERT.

[0077] The following is an example to illustrate this point. As shown in Figure 5, it is a schematic diagram of the training of the backbone network. In Figure 5, after the Natural Language Description (i.e., style description text) is input to BERT, Style Embedding (style vectors) is obtained. The Input text (i.e., input text) is input to the Slow Transformer + Fast Transformer (encoding network in the backbone network), and the predicted style sequences in the four style dimensions output by the Slow Transformer (i.e., pitch, volume, duration, emotion in Figure 5) are obtained. The predicted style sequences in the four style dimensions are combined to perform parameter tuning on the backbone network.

[0078] Here, the Slow Transformer determines the predicted style sequence in the four style dimensions based on the processed Hidden States (intermediate feature vectors), Style Embedding, and hierarchical cross-attention mechanism. The Fast Transformer then performs further feature extraction and speech synthesis based on the processed intermediate feature vectors, Style Embedding, and hierarchical cross-attention mechanism. Here, the duration in Figure 5 can represent the speech rate sequence.

[0079] To realize the above embodiments, the present disclosure further provides a speech synthesis model training apparatus. As shown in Figure 6, Figure 6 is a schematic diagram of a fourth embodiment of the present disclosure. The speech synthesis model training apparatus 60 may include a first acquisition module 601, a second acquisition module 602, and a training processing module 603.

[0080] Here, the first acquisition module 601 acquires training data, and the training samples in the training data include style description sample text, input sample text, output sample audio, and style dimension data of the output sample audio. The second acquisition module 602 acquires an initial speech synthesis model, and the training processing module 603 performs training processing on the speech synthesis model based on the style description sample text, input sample text, output sample audio, and style dimension data to acquire a trained speech synthesis model.

[0081] As one possible implementation of the embodiments of this disclosure, the speech synthesis model includes a backbone network and a style control network, wherein the style control network performs style extraction processing on an input style description text to obtain a style feature vector, and the backbone network performs speech synthesis processing by combining the input text and the style feature vector to obtain output speech.

[0082] In one possible implementation of an embodiment of the present disclosure, the style dimension data includes sample style content in at least one style dimension of the output sample audio, and the training processing module 603 specifically trains the style control network in the speech synthesis model based on the style description sample text and the sample style content in at least one style dimension of the output sample audio to obtain a trained style control network, and trains the backbone network in the speech synthesis model based on the style description sample text, the input sample text, the output sample audio and the trained style control network to obtain a trained backbone network.

[0083] As one possible implementation of an embodiment of the present disclosure, the training processing module 603 further inputs the style description sample text into the style control network to obtain a first predicted style vector output by the style control network, determines the predicted style content in at least one style dimension based on the first predicted style vector, and performs parameter tuning on the style control network based on the predicted style content and the sample style content in at least one style dimension to obtain the trained style control network.

[0084] In one possible embodiment of the embodiments of the present disclosure, the style dimension data further includes a sample style sequence in at least one style dimension of the output sample audio, the sample style sequence includes the style content of each audio frame in the output sample audio, the training processing module 603 further inputs the style description sample text into the trained style control network to obtain a second predicted style vector output by the style control network, the second predicted style vector and the input sample text into the backbone network to obtain a predicted style sequence in at least one style dimension and an output predicted audio output by the backbone network, and performs parameter tuning on the backbone network based on the predicted style sequence in at least one style dimension, the sample style sequence, the output predicted audio and the output sample audio to obtain the trained backbone network.

[0085] In one possible embodiment of the embodiments of the present disclosure, the training processing module 603 further determines a first loss value based on the predicted style sequence and the sample style sequence in at least one style dimension, determines a second loss value based on the output predicted speech and the output sample speech, and performs parameter tuning on the backbone network based on the first loss value and the second loss value to obtain the trained backbone network.

[0086] In one possible implementation of an embodiment of the present disclosure, the backbone network includes an encoding network and a decoding network, wherein the encoding network and / or the decoding network are provided with a hierarchical cross-attention mechanism for determining a predicted style sequence in at least one style dimension based on an intermediate feature vector and the second predicted style vector, the intermediate feature vector being an intermediate vector in the processing steps of the encoding network and / or the decoding network.

[0087] In one possible implementation of an embodiment of the present disclosure, the style dimension data includes sample style content and sample style sequences in at least one style dimension of the output sample audio, and the first acquisition module specifically acquires the style description sample text, the input sample text and the output sample audio, performs a style extraction process in at least one style dimension for each audio frame in the output sample audio to acquire a sample style sequence in at least one style dimension, and determines the sample style content in at least one style dimension based on the sample style sequence in at least one style dimension.

[0088] In one possible embodiment of the embodiments of the present disclosure, the style dimension includes at least one of pitch, volume, speech rate, pitch variance, and emotion type.

[0089] The speech synthesis model training apparatus of the embodiment of this disclosure acquires training data, the training samples in the training data include style description sample text, input sample text, output sample audio, and style dimension data of the output sample audio, acquires an initial speech synthesis model, and performs training processing on the speech synthesis model based on the style description sample text, input sample text, output sample audio, and style dimension data to acquire a trained speech synthesis model. Here, by performing training processing on the speech synthesis model in combination with style dimension data, the electronic device can take into account many features during the training of the speech synthesis model, thereby improving the accuracy of the trained speech synthesis model.

[0090] To realize the above embodiments, the present disclosure further provides a speech synthesis device. As shown in Figure 7, Figure 7 is a schematic diagram of a fifth embodiment of the present disclosure. The speech synthesis device 70 may include a first acquisition module 701, a second acquisition module 702, and a third acquisition module 703.

[0091] Here, the first acquisition module 701 acquires the input text and the style description text, the second acquisition module 702 inputs the style description text into the style control network in the speech synthesis model and acquires the style vector output by the style control network, the speech synthesis model is obtained by training on the style description sample text, the input sample text, the output sample speech and the style dimension data of the output sample speech, and the third acquisition module 703 inputs the style vector and the input text into the backbone network in the speech synthesis model and acquires the output speech corresponding to the input text output by the backbone network.

[0092] The speech synthesis apparatus of the embodiment of this disclosure acquires input text and style description text, inputs the style description text into a style control network in the speech synthesis model, and obtains a style vector output by the style control network. The speech synthesis model is obtained by training it based on style description sample text, input sample text, output sample speech, and style dimension data of the output sample speech. The style vector and input text are input into a backbone network in the speech synthesis model, and the output speech corresponding to the input text is obtained from the backbone network. Here, since style-related features can be considered in the speech synthesis model, the degree of style matching between the determined output speech and the style description text can be increased, and the accuracy of the output speech can be further improved.

[0093] In the proposed technology disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of relevant users' personal information will be carried out with the user's consent, in accordance with the provisions of relevant laws and regulations, and will not violate public order and morals.

[0094] According to embodiments of the present disclosure, the present disclosure further provides electronic devices and readable storage media. According to embodiments of the present disclosure, the present disclosure further provides a computer program which, when executed by a processor, is performed, a method for training a speech synthesis model or a speech synthesis method provided by embodiments of the present disclosure.

[0095] Figure 8 shows a schematic block diagram of an exemplary electronic device 800 that can carry out embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, mobile phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the description herein and / or the implementation of the application as required.

[0096] As shown in Figure 8, device 800 includes a computing unit 801 capable of performing various appropriate operations and processes based on computer programs stored in read-only memory (ROM) 802 or computer programs loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 can contain various programs and data necessary for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are connected to each other via bus 804. An input / output (I / O) interface 805 is similarly connected to bus 804.

[0097] Multiple components within device 800, including input units 806 such as a keyboard and mouse, output units 807 such as monitors and speakers of various types, storage units 808 such as magnetic disks and optical disks, and communication units 809 such as a network card, modem, and wireless communication transmitter / receiver, are connected to the I / O interface 805. The communication unit 809 allows device 800 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunications networks.

[0098] The computing unit 801 may be a variety of general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), a variety of dedicated artificial intelligence (AI) computing chips, a variety of computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs each of the methods and processes described above, for example, a method for training a speech synthesis model or a method for synthesizing speech. For example, in some embodiments, the method for training a speech synthesis model or a method for synthesizing speech can be implemented as a computer software program tangibly contained in a machine-readable medium such as a memory unit 808. In some embodiments, part or all of the computer program is loaded and / or installed into device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the method for training a speech synthesis model or a method for synthesizing speech described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the speech synthesis model training method or speech synthesis method via any other suitable method (e.g., via firmware).

[0099] Various embodiments of the systems and technologies described herein can be implemented as digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be implemented as one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be an application-specific or general-purpose programmable processor, which can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0100] Program code for performing the method of this application can be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing device, so that when executed by the processor or controller, the functions / operations defined by the flowchart and / or block diagrams are performed. The program code may run entirely on a machine, partially on a machine, as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0101] In the context of this application, a machine-readable medium may be a tangible medium that contains or can store a program used by or in combination with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include one or more line-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0102] To provide user interaction, the systems and technologies described herein can be implemented on a computer, which may have a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), as well as a keyboard and pointing device (e.g., a mouse or trackball), and the user may provide input to the computer via the keyboard and pointing device. Other types of devices may also provide user interaction; for example, the feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and may receive input from the user in any form (including acoustic input and voice input or haptic input).

[0103] The systems and technologies described herein can be implemented in a computing system including backend components (e.g., as a data server), a computing system including middleware components (e.g., an application server), a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser, through which the user interacts with embodiments of the systems and technologies described herein), or in a computing system including any combination of such backend components, middleware components, and frontend components. The components of the system can be interconnected by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the internet, and blockchain networks.

[0104] A computer system can include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is generated by computer programs running on corresponding computers that have a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0105] Furthermore, the steps can be rearranged, added, or deleted using the various forms of flows shown above. For example, each step described in this application may be performed in parallel, sequentially, or in a different order, as long as the desired results of the proposed technology disclosed herein can be achieved.

[0106] The specific embodiments described above do not limit the scope of protection of this disclosure. Those skilled in the art will understand that various modifications, combinations, partial combinations, and substitutions can be made depending on the design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training speech synthesis models, A step of acquiring training data, wherein the training samples in the training data include a style description sample text, an input sample text, an output sample audio, and style dimension data of the output sample audio. Steps to obtain an initial speech synthesis model, The steps include: performing a training process on the speech synthesis model based on the style description sample text, the input sample text, the output sample audio, and the style dimension data to obtain a trained speech synthesis model; A method for training speech synthesis models, including [specific details omitted].

2. The aforementioned speech synthesis model includes a backbone network and a style control network. The aforementioned style control network is used to perform style extraction processing on the input style description text and obtain style feature vectors. The method for training a speech synthesis model according to claim 1, wherein the backbone network is used to perform speech synthesis processing by combining the input text and the style feature vector to obtain output speech.

3. The style dimension data includes sample style content in at least one style dimension of the output sample audio. The step of performing a training process on the speech synthesis model based on the style description sample text, the input sample text, the output sample audio, and the style dimension data to obtain a trained speech synthesis model is: The steps include: performing a training process on the style control network in the speech synthesis model based on the style description sample text and the sample style content in at least one style dimension of the output sample audio, and obtaining a trained style control network; The steps include: performing a training process on the backbone network in the speech synthesis model based on the style description sample text, the input sample text, the output sample speech, and the trained style control network, and obtaining a trained backbone network; A method for training a speech synthesis model according to claim 1, including the following:

4. The step of training the style control network in the speech synthesis model based on the style description sample text and the sample style content in at least one style dimension of the output sample speech, and obtaining the trained style control network, is: The steps include inputting the aforementioned style description sample text into the style control network and obtaining a first predicted style vector output by the style control network, A step of determining the predicted style content in at least one style dimension based on the first predicted style vector, A step of obtaining the trained style control network by performing a parameter tuning process on the style control network based on the predicted style content and the sample style content in at least one style dimension, A method for training a speech synthesis model according to claim 3, including the following:

5. The style dimension data further includes a sample style sequence in at least one style dimension of the output sample audio, the sample style sequence includes the style content of each audio frame in the output sample audio, The step of training the backbone network in the speech synthesis model based on the style description sample text, the input sample text, the output sample speech, and the trained style control network, in order to obtain a trained backbone network, is: The steps include inputting the aforementioned style description sample text into the trained style control network and obtaining a second predicted style vector output by the style control network, The steps include inputting the second predicted style vector and the input sample text into the backbone network to obtain a predicted style sequence in at least one style dimension and an output predicted speech from the backbone network, A step of obtaining the trained backbone network by performing parameter tuning on the backbone network based on the predicted style sequence, the sample style sequence, the predicted output speech, and the output sample speech in at least one style dimension, A method for training a speech synthesis model according to claim 3, including the following:

6. The step of obtaining the trained backbone network by performing parameter tuning on the backbone network based on the predicted style sequence, the sample style sequence, the predicted output speech, and the output sample speech in at least one style dimension is: A step of determining a first loss value based on the predicted style sequence and the sample style sequence in at least one style dimension, The steps include determining a second loss value based on the output prediction audio and the output sample audio, The steps include: performing parameter adjustment processing on the backbone network based on the first loss value and the second loss value to obtain the trained backbone network; A method for training a speech synthesis model according to claim 5, including the following:

7. The backbone network includes a coding network, and the coding network is provided with a hierarchical cross-attention mechanism for determining a predicted style sequence in at least one style dimension based on an intermediate feature vector and the second predicted style vector. The method for training a speech synthesis model according to claim 5, wherein the intermediate feature vector is an intermediate vector in the processing process of the coding network.

8. The style dimension data includes sample style content and sample style sequence in at least one style dimension of the output sample audio. The step of acquiring the aforementioned training data is: The steps include obtaining the aforementioned style description sample text, the aforementioned input sample text, and the output sample audio, The steps include: performing a style extraction process in at least one style dimension on each audio frame in the output sample audio to obtain a sample style sequence in at least one style dimension; A step of determining sample style content in at least one style dimension based on a sample style sequence in at least one style dimension, A method for training a speech synthesis model according to claim 1, including the following:

9. The method for training a speech synthesis model according to claim 2, wherein the style dimension includes at least one of pitch, volume, speech rate, pitch variance, and emotion type.

10. A speech synthesis method, Steps to obtain input text and style description text, A step of inputting the style description text into a style control network in a speech synthesis model and obtaining a style vector output by the style network, wherein the speech synthesis model is obtained by training it based on a style description sample text, an input sample text, an output sample speech, and the style dimension data of the output sample speech. The steps include inputting the style vector and the input text into the backbone network of the speech synthesis model to obtain the output speech corresponding to the input text output by the backbone network, A speech synthesis method that includes [this].

11. A training device for speech synthesis models, A first acquisition module for acquiring training data, wherein the training samples in the training data include a style description sample text, an input sample text, an output sample audio, and style dimension data of the output sample audio. A second acquisition module for obtaining the initial speech synthesis model, A training processing module for performing training on the speech synthesis model based on the style description sample text, the input sample text, the output sample audio, and the style dimension data, and obtaining a trained speech synthesis model. A training device for speech synthesis models, including one that includes this feature.

12. A speech synthesis device, A first acquisition module for obtaining input text and style description text, A second acquisition module for inputting the style description text into a style control network in a speech synthesis model and obtaining a style vector output by the style control network, wherein the speech synthesis model is obtained by training on a style description sample text, an input sample text, an output sample speech, and the style dimension data of the output sample speech, A third acquisition module for inputting the style vector and the input text into the backbone network of the speech synthesis model and acquiring the output speech corresponding to the input text output by the backbone network, A speech synthesis device that includes [this component].

13. It is an electronic device, At least one processor, Includes a memory that is communicably connected to at least one of the processors, An electronic device wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, thereby causing the at least one processor to perform the method according to any one of claims 1 to 9 or the method according to claim 10.

14. A non-temporary computer-readable storage medium storing computer instructions, wherein the computer instructions cause a computer to execute the method according to any one of claims 1 to 9 or the method according to claim 10.

15. A computer program, wherein, when the computer program is executed by a processor, the method according to any one of claims 1 to 9 or the method according to claim 10 is implemented.

Images