Training methods for speech synthesis models, speech synthesis methods and devices

Abstract

This application proposes a training method, a speech synthesis method, and an apparatus for a speech synthesis model, relating to artificial intelligence technologies such as deep learning and speech technology. The training method for the speech synthesis model includes: acquiring sample acoustic features, corresponding first phoneme sequences, and corresponding speaker identifiers of multiple first sample audio samples, wherein at least one first sample audio sample corresponding to the same speaker identifier has a single style feature; inputting the first phoneme sequence and corresponding speaker identifier of the first sample audio sample into the speech synthesis model to obtain the predicted acoustic features of the first sample audio sample; and training the speech synthesis model based on the predicted acoustic features and sample acoustic features of each first sample audio sample. This achieves the decoupling of timbre features and style features in the audio, thereby enabling the use of audio samples with single style features corresponding to multiple speakers to train the speech synthesis model, thus reducing the training cost of the speech synthesis model.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence technology such as deep learning and speech technology, and in particular to a training method for a speech synthesis model, a speech synthesis method, and an apparatus. Background Technology

[0002] Currently, speech synthesis technology is widely used in scenarios such as intelligent question answering, voice broadcasting, audiobooks, and virtual anchors. In some scenarios, it is necessary to synthesize audio from the same speaker in different styles.

[0003] In order to synthesize audio from the same speaker in different styles, related technologies require using audio recordings of different styles from each speaker as training data to train and generate a speech synthesis model. Due to the high cost of audio recording, the training cost of the speech synthesis model is also high. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] This application proposes a training method, a speech synthesis method, and an apparatus for a speech synthesis model, in order to solve the technical problem of high training cost of speech synthesis models in related technologies.

[0006] The first aspect of this application proposes a training method for a speech synthesis model, comprising: acquiring sample acoustic features, corresponding first phoneme sequences, and corresponding speaker identifiers of multiple first sample audios, wherein at least one first sample audio corresponding to the same speaker identifier has a single style feature; inputting the first phoneme sequence corresponding to the first sample audio and the corresponding speaker identifier into the encoding layer of the speech synthesis model to determine the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and determining the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features; inputting the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer of the speech synthesis model to decode based on the timbre representation corresponding to the speaker identifier and the text encoding to obtain the predicted acoustic features of the first sample audio; and training the speech synthesis model based on the predicted acoustic features of each first sample audio and the sample acoustic features.

[0007] A second aspect of this application proposes a speech synthesis method, comprising: acquiring a third phoneme sequence corresponding to a target text to be synthesized, and acquiring a first speaker identifier and a second speaker identifier from a candidate identifier set; inputting the third phoneme sequence and the first speaker identifier into the encoding layer of a speech synthesis model to determine the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence based on the style representation corresponding to the first speaker identifier and the third phoneme sequence, and determining the text encoding of the third phoneme sequence at the audio frame level based on the third phoneme sequence and the prosodic features; inputting the text encoding of the third phoneme sequence at the audio frame level and the second speaker identifier into the decoding layer of the speech synthesis model to decode based on the timbre representation corresponding to the second speaker identifier and the text encoding to obtain acoustic features; and generating target audio corresponding to the target text based on the acoustic features.

[0008] A third aspect of this application proposes a model training apparatus for speech synthesis, comprising: a first acquisition module, configured to acquire sample acoustic features, corresponding first phoneme sequences, and corresponding speaker identifiers of multiple first sample audios, wherein at least one first sample audio corresponding to the same speaker identifier has a single style feature; a first processing module, configured to input the first phoneme sequence corresponding to the first sample audio and the corresponding speaker identifier into the encoding layer of a speech synthesis model, to determine the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and to determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features; a second processing module, configured to input the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer of the speech synthesis model, to decode based on the timbre representation corresponding to the speaker identifier and the text encoding to obtain the predicted acoustic features of the first sample audio; and a first training module, configured to train the speech synthesis model based on the predicted acoustic features of each first sample audio and the sample acoustic features.

[0009] A fourth aspect of this application provides a speech synthesis apparatus, comprising: a third acquisition module, configured to acquire a third phoneme sequence corresponding to a target text to be synthesized, and acquire a first speaker identifier and a second speaker identifier from a candidate identifier set; a third processing module, configured to input the third phoneme sequence and the first speaker identifier into the encoding layer of a speech synthesis model, to determine the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence based on the style representation corresponding to the first speaker identifier and the third phoneme sequence, and to determine the text encoding of the third phoneme sequence at the audio frame level based on the third phoneme sequence and the prosodic features; a fourth processing module, configured to input the text encoding of the third phoneme sequence at the audio frame level and the second speaker identifier into the decoding layer of the speech synthesis model, to decode based on the timbre representation corresponding to the second speaker identifier and the text encoding to obtain acoustic features; and a generation module, configured to generate target audio corresponding to the target text based on the acoustic features.

[0010] A fifth aspect of this application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a training method for a speech synthesis model as proposed in a first aspect of this application, or to perform a speech synthesis method as proposed in a second aspect of this application.

[0011] A sixth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform a training method for a speech synthesis model as proposed in a first aspect of this application, or to perform a speech synthesis method as proposed in a second aspect of this application.

[0012] A seventh aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements a training method for a speech synthesis model as proposed in a first aspect of this application, or executes a speech synthesis method as proposed in a second aspect of this application.

[0013] One embodiment of the above invention has the following advantages or beneficial effects:

[0014] This approach decouples timbre and style features in audio, allowing the training of a speech synthesis model using audio samples from multiple speakers, each with a unique style. This reduces the cost of recording audio data for training, and consequently, the training cost of the speech synthesis model. Furthermore, the same speech synthesis model can be used to flexibly generate target audio samples with different timbres and styles, enhancing the model's flexibility, expanding its applicability, and reducing the number of models required for style transfer.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0017] Figure 1 This is a flowchart illustrating the training method for the speech synthesis model provided in Embodiment 1 of this application;

[0018] Figure 2 This is a flowchart illustrating the training method for the speech synthesis model provided in Embodiment 2 of this application;

[0019] Figure 3 This is a schematic diagram of the speech synthesis model provided in Embodiment 2 of this application;

[0020] Figure 4 This is a schematic diagram of the prosody prediction module provided in Embodiment 2 of this application;

[0021] Figure 5 This is a flowchart illustrating the speech synthesis method provided in Embodiment 3 of this application;

[0022] Figure 6 This is a schematic diagram of the pitch characteristics of target audio a and target audio b provided in Embodiment 3 of this application;

[0023] Figure 7 This is a schematic diagram of the pitch characteristics of target audio c and target audio d provided in Embodiment 3 of this application;

[0024] Figure 8 This is a schematic diagram of the pitch characteristics of the target audio e and the target audio f provided in Embodiment 3 of this application;

[0025] Figure 9 This is a schematic diagram of the structure of the training device for the speech synthesis model provided in Embodiment 4 of this application;

[0026] Figure 10This is a schematic diagram of the speech synthesis device provided in Embodiment 5 of this application;

[0027] Figure 11 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown. Detailed Implementation

[0028] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0029] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0030] This application addresses the technical problem of high training costs for speech synthesis models in related technologies by proposing a training method for a speech synthesis model, a speech synthesis method, a device, an electronic device, a storage medium, and a computer program product.

[0031] The training method for the speech synthesis model includes: acquiring the acoustic features of multiple first sample audio samples, their corresponding first phoneme sequences, and their corresponding speaker identifiers, wherein at least one first sample audio sample corresponding to the same speaker identifier has a single style feature; inputting the first phoneme sequence corresponding to the first sample audio sample and the corresponding speaker identifier into the encoding layer of the speech synthesis model, to determine the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and to determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features; inputting the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer of the speech synthesis model, to decode based on the timbre representation corresponding to the speaker identifier and the text encoding to obtain the predicted acoustic features of the first sample audio sample; and training the speech synthesis model based on the predicted acoustic features and sample acoustic features of each first sample audio sample. This decouples timbre and style features in audio, allowing the speech synthesis model to be trained using audio samples from multiple speakers, each with a single style feature. This reduces the cost of recording audio data for training, and consequently, the training cost of the speech synthesis model.

[0032] To facilitate understanding, the technical terms involved in the embodiments of this application will be explained first.

[0033] In the description of the present application, the timbre feature is a feature of the speaker's timbre, which may include, for example, but not limited to, features such as the wavelength, frequency, intensity, and rhythm of the speaker's voice. Different speakers have different timbre features.

[0034] The style feature is used to characterize the speaking style, speaking characteristics, or language expressiveness of the speaker. For example, the style feature may include, but not limited to, features such as the light reading, weak reading, strong reading, lengthening, and emphasis of each word or term when the speaker speaks.

[0035] The speaker identifier is used to uniquely identify the speaker. Among them, the speaker identifier may be a number pre-set for the speaker, or an identifier such as the speaker's name, and the present application does not limit this.

[0036] The phoneme sequence is a sequence composed of multiple phonemes. Among them, a phoneme is the smallest speech unit divided according to the natural attributes of speech. For example, taking the text "they" as an example, the corresponding phoneme sequence is the phoneme sequence "t a m en" composed of the phonemes "t", "a", "m", and "en". It should be noted that in the embodiments of the present application, the numbers corresponding to each phoneme may be pre-set, so that the phonemes can be represented by the corresponding numbers, and a phoneme sequence composed of the numbers corresponding to each phoneme can be obtained.

[0037] The acoustic feature characterizes the acoustic characteristics of speech. For example, it can be the Mel spectrum feature.

[0038] The style representation is used to characterize the style feature of the corresponding speaker, and can be the style embedding vector corresponding to the speaker.

[0039] The timbre representation is used to characterize the timbre feature of the corresponding speaker, and can be the timbre embedding vector corresponding to the speaker.

[0040] The prosodic features of the speaker on each phoneme in a certain phoneme sequence are used to characterize the pronunciation style of the speaker on each phoneme, and may include features such as pitch, duration, and energy.

[0041] The audio frame can be 10 ms (milliseconds) or other lengths, which can be set as needed, and the present application does not limit this.

[0042] The text encoding of a phoneme sequence at the audio frame level includes the text encoding corresponding to multiple audio frames into which the phoneme sequence is divided according to the audio frame level. The text encoding corresponding to each audio frame contains the features of the text corresponding to that audio frame and the speaker's prosodic features in that audio frame. Among them, the speaker's prosodic features in that audio frame are used to characterize the speaker's pronunciation style in that audio frame, and may include features such as pitch, duration, and energy.

[0043] The following description, with reference to the accompanying drawings, describes a training method for a speech synthesis model, a speech synthesis method, an apparatus, an electronic device, a storage medium, and a computer program product according to embodiments of this application.

[0044] First, the training method of the speech synthesis model provided in the embodiments of this application will be described.

[0045] It should be noted that the training method for the speech synthesis model provided in this application embodiment is executed by a speech synthesis model training device. This speech synthesis model training device can be an electronic device or configured within an electronic device. By executing the speech synthesis model training method provided in this application embodiment, it achieves decoupling of timbre and style features in the audio. This allows the speech synthesis model to be trained using audio samples from multiple speakers, each with a single style feature, thereby reducing the audio recording cost in the training data and consequently reducing the training cost of the speech synthesis model.

[0046] Among them, the electronic device can be a personal computer (PC), cloud device, mobile device, server, etc. The mobile device can be any hardware device such as mobile phone, tablet computer, personal digital assistant, wearable device, vehicle device, etc. This application does not limit it.

[0047] Figure 1 This is a flowchart illustrating the training method for the speech synthesis model provided in Embodiment 1 of this application. Figure 1 As shown, the training method for this speech synthesis model may include the following steps 101-104.

[0048] Step 101: Obtain the acoustic features of multiple first sample audios, the corresponding first phoneme sequences, and the corresponding speaker identifiers, wherein at least one first sample audio corresponding to the same speaker identifier has a single style feature.

[0049] Among them, the acoustic features of the first sample audio represent the acoustic characteristics of the first sample audio, such as Mel spectrum features, which are obtained by extracting acoustic features from the first sample audio.

[0050] The first phoneme sequence is a phoneme sequence obtained by performing phoneme conversion on the text corresponding to the first sample audio. For example, taking the text "they" corresponding to the first sample audio as an example, phoneme conversion can be performed on this text to obtain the first phoneme sequence "t a m en" corresponding to the first sample audio. Among them, "t", "a", "m", and "en" are phonemes.

[0051] The same speaker identifier can correspond to one first sample audio or multiple first sample audios, and this application does not limit this.

[0052] The first sample audios corresponding to at least one same speaker identifier have a single style feature, that is, at least one speaker can record only audios with one style feature as training data.

[0053] It should be noted that the above training data when training the speech synthesis model in the embodiments of this disclosure can be provided by users and authorized for use, or obtained from public datasets, or obtained through other means that comply with the relevant laws and regulations. This application does not limit this.

[0054] Step 102: Input the first phoneme sequence corresponding to the first sample audio and the corresponding speaker identifier into the encoding layer of the speech synthesis model, so as to determine the prosody features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosody features.

[0055] Among them, the speech synthesis model is a neural network model with speech synthesis capabilities.

[0056] The style representation corresponding to the speaker identifier is used to represent the style features of the speaker corresponding to the speaker identifier and can be the Style embedding corresponding to the speaker identifier.

[0057] The prosody features of the speaker on each phoneme in the first phoneme sequence are used to represent the pronunciation style of the speaker on each phoneme in the first phoneme sequence and can include features such as pitch, duration, and energy.

[0058] The text encoding of the first phoneme sequence at the audio frame level includes the text encodings corresponding to multiple audio frames obtained by dividing the first phoneme sequence at the audio frame level. The text encoding corresponding to each audio frame contains the features of the text corresponding to the audio frame and the prosody features of the speaker on the audio frame.

[0059] In one embodiment of this application, the speech synthesis model may include an encoding layer. The first phoneme sequence corresponding to the same first sample audio and the corresponding speaker identifier may be input into the encoding layer of the speech synthesis model. The encoding layer may query the embedding table based on the speaker identifier to obtain the style representation corresponding to the speaker identifier. Then, based on the style representation corresponding to the speaker identifier and the first phoneme sequence, the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence may be determined. Based on the first phoneme sequence and the prosodic features, the text encoding of the first phoneme sequence at the audio frame level may be determined.

[0060] Step 103: Input the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer of the speech synthesis model, and decode it based on the timbre representation and text encoding corresponding to the speaker identifier to obtain the predicted acoustic features of the first sample audio.

[0061] Among them, the timbre representation corresponding to the speaker identifier is used to represent the timbre features of the speaker corresponding to the speaker identifier, and can be the speaker embedding corresponding to the speaker identifier.

[0062] Predict acoustic features to characterize the acoustic properties of the first sample audio, such as the Mel spectrum, which is predicted by the speech synthesis model.

[0063] In one embodiment of this application, the speech synthesis model may include a decoding layer. After inputting the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer, the decoding layer can query an embedding table based on the speaker identifier to obtain the timbre representation corresponding to the speaker identifier. Then, it can perform decoding based on the timbre representation and the text encoding to obtain predicted acoustic features. These predicted acoustic features carry the text features, speaker style features, and speaker timbre features of the text corresponding to the first sample audio.

[0064] It should be noted that in the embodiments of this application, different representation tables are used to obtain the corresponding timbre representation and style representation for the same speaker identifier. Therefore, for the same speaker, although the speaker identifier is the same, the timbre representation and style representation are different.

[0065] Step 104: Train the speech synthesis model based on the predicted acoustic features and sample acoustic features of each first sample audio.

[0066] In the embodiments of this application, the predicted acoustic features of each first sample audio and the sample acoustic features of the corresponding first sample audio can be substituted into the loss function to determine the loss value. Then, the model parameters of the speech synthesis model are adjusted according to the loss value. Through multiple iterations of optimization, the trained speech synthesis model is obtained.

[0067] The loss function can be set as needed, such as the mean squared error (MSE) loss function or other loss functions. This application does not impose any restrictions on this.

[0068] In this embodiment, since the coding layer of the speech synthesis model does not rely on timbre representation when determining the text encoding of the first phoneme sequence at the audio frame level, and timbre representation is only used by the decoding layer to determine the predicted acoustic features, only the decoding layer and timbre representation affect the timbre features carried in the predicted acoustic features, thus achieving decoupling between text and timbre features. Furthermore, when the coding layer relies on style representation to determine the text encoding of the first phoneme sequence at the audio frame level, it does not rely on timbre representation, and the prosodic features determined based on style representation are only related to style features and not to timbre features. Therefore, when the coding layer relies on style features to determine the text encoding of the first phoneme sequence at the audio frame level, it does not rely on timbre features, thus achieving decoupling between style features and timbre features.

[0069] Because speech synthesis models decouple timbre and style features in audio, training data can be generated using audio of a single speaker style. During training, each speaker's data is treated as an independent style, ensuring each speaker has a distinct style. Based on the speaker's identifier, prosodic features of different styles can be predicted. Since a single speaker can use audio of a single style for training, the requirements for training data are reduced, eliminating the need for each speaker to record audio of different styles. This reduces the recording cost of training audio and consequently lowers the training cost of the speech synthesis model.

[0070] Furthermore, by training the speech synthesis model in the aforementioned manner, the model learns the individual timbre and style features of each speaker during training. Therefore, using the speech synthesis model trained in this embodiment, it is possible to synthesize target audio with the timbre features corresponding to one speaker identifier and the style features corresponding to the other speaker identifier, based on arbitrary text data and any two speaker identifiers. Thus, the same speech synthesis model can be used to flexibly generate target audio with different timbres and styles, improving the flexibility of the speech synthesis model, expanding its applicability, and reducing the number of models required for style transfer.

[0071] In summary, the training method for the speech synthesis model provided in this application involves obtaining sample acoustic features, corresponding first phoneme sequences, and corresponding speaker identifiers for multiple first sample audios, where at least one first sample audio corresponding to the same speaker identifier has a single style feature; inputting the first phoneme sequence and corresponding speaker identifier of the first sample audio into the encoding layer of the speech synthesis model to determine the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and determining the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and prosodic features; inputting the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer of the speech synthesis model to decode based on the timbre representation and text encoding corresponding to the speaker identifier to obtain the predicted acoustic features of the first sample audio; and training the speech synthesis model based on the predicted acoustic features and sample acoustic features of each first sample audio. This decouples timbre and style features in audio, allowing the speech synthesis model to be trained using audio samples from multiple speakers, each with a single style feature. This reduces the cost of recording audio data for training, and consequently, the training cost of the speech synthesis model.

[0072] The following is combined with Figure 2 The process of determining the predicted acoustic features of the first sample audio through the speech synthesis model in the training method of the speech synthesis model in the embodiments of this application will be further explained.

[0073] Figure 2 This is a schematic flowchart illustrating the training method for the speech synthesis model provided in Embodiment 2 of this application. Figure 2 As shown, the training method for the speech synthesis model may include the following steps 201-209.

[0074] Step 201: Obtain the acoustic features of multiple first sample audios, the corresponding first phoneme sequences, and the corresponding speaker identifiers, wherein at least one first sample audio corresponding to the same speaker identifier has a single style feature.

[0075] The specific implementation process and principle of step 201 can be referred to the description of the above embodiments, and will not be repeated here.

[0076] Step 202: Input the speaker identifier into the second embedding module to obtain the style representation corresponding to the speaker identifier.

[0077] Step 203: Input the first phoneme sequence into the first embedding module to obtain the text representation of each phoneme in the first phoneme sequence, and input the text representation of each phoneme into the encoder to obtain the text encoding of each phoneme.

[0078] Step 204: Input the text encoding and style representation of each phoneme into the prosody prediction module to obtain the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence, and determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features.

[0079] In one embodiment of this application, reference is made to Figure 3 The coding layer in the speech synthesis model may include a first embedding module 31, an encoder 32, a prosody prediction module 33, and a second embedding module 34 connected in sequence. The encoder 32 may be a multi-layer feedforward neural network (FFN) based on a self-attention mechanism.

[0080] Correspondingly, by inputting the speaker identifier into the second embedding module 34, the style representation corresponding to the speaker identifier can be obtained through the second embedding module. By inputting the first phoneme sequence into the first embedding module 31, the text representation of each phoneme in the first phoneme sequence can be obtained through the first embedding module 31. The text representation, used to represent the corresponding phoneme, can be the phoneme embedding vector. Further, the text representation of each phoneme in the first phoneme sequence and the corresponding position embedding vector can be input into the encoder 32 to obtain the text encoding of each phoneme. The text encoding is used to represent the feature information of the corresponding phoneme. Further, the text encoding and style representation of each phoneme can be input into the prosody prediction module 33 to obtain the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence. The prosody prediction module 33 then determines the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features.

[0081] In one embodiment of this application, reference is made to Figure 3 The prosody prediction module 33 may include a prosody prediction unit 331 and a prosody processing unit 332. Prosody features include pitch features, energy features, and duration features. Accordingly, step 204 can be implemented as follows: inputting the text encoding and style representation of each phoneme into the prosody prediction unit 331 to obtain the pitch features, energy features, and duration features of the speaker corresponding to the speaker identifier on each phoneme; inputting the text encoding of each phoneme and the speaker's pitch features, energy features, and duration features on each phoneme into the prosody processing unit 332 to fuse the speaker's pitch features and energy features on each phoneme with the text encoding of each phoneme to obtain the fused encoding of each phoneme; and through the prosody processing unit 332, based on the speaker's duration features on each phoneme, extending the fused encoding of each phoneme to the audio frame level to obtain the text encoding of the first phoneme sequence at the audio frame level.

[0082] The prosodic prediction unit 331 may include multiple prosodic prediction subunits, each of which is used to predict a prosodic feature based on the text encoding and style representation of each phoneme.

[0083] For example, the structures of the prosody prediction unit 331 and the prosody processing unit 332 can be... Figure 4 The structure shown. Where, reference Figure 4 The prosody prediction unit 331 may include three prosody prediction subunits: a duration prediction subunit 3311, a pitch prediction subunit 3312, and an energy prediction subunit 3313. The prosody processing unit 332 may include a length adjustment subunit 3321. Figure 4 The "+" sign indicates that corresponding elements of the vectors are added together. The structure of each prosodic prediction subunit can be found in relevant technologies and will not be elaborated upon here.

[0084] In one embodiment of this application, reference is made to Figure 4 The text encoding and style representation of each phoneme can be input into the duration prediction subunit 3311, pitch prediction subunit 3312, and energy prediction subunit 3313. Thus, the duration prediction subunit 3311 can determine the duration feature of each phoneme based on its text encoding and style representation; the pitch prediction subunit 3312 can determine the pitch feature of each phoneme based on its text encoding and style representation; and the energy prediction subunit 3313 can determine the energy feature of each phoneme based on its text encoding and style representation. The pitch and energy features of each phoneme can be fused with its text encoding to obtain a fused encoding. Furthermore, the fused encoding and duration features of each phoneme can be input into the length adjustment subunit 3321, which extends the fused encoding of each phoneme to the audio frame level, obtaining the text encoding of the first phoneme sequence at the audio frame level. Among them, the pitch features, energy features, and duration features of phonemes can be added to the backbone network where the text encoding is located through one-dimensional convolution.

[0085] In one embodiment of this application, the pitch and energy features of each phoneme in the first phoneme sequence can be Gaussian normalized for each speaker, so that the pitch and energy features do not contain any speaker information and only represent the overall prosodic expression. The duration features of the phonemes can be obtained through a forced alignment method.

[0086] Step 205: Input the speaker identifier into the third embedding module to obtain the timbre representation corresponding to the speaker identifier.

[0087] Step 206: Input the timbre representation and the text encoding of the first phoneme sequence at the audio frame level into the decoder to decode based on the timbre representation and the text encoding to obtain the predicted acoustic features.

[0088] In one embodiment of this application, reference is made to Figure 3 The decoding layer of the speech synthesis model includes a decoder 35 connected to the encoding layer (specifically, the prosodic processing unit 332 in the encoding layer) and a third embedding module 36 connected to the decoder 35. The decoder 35 can be a multi-layer FFN network based on a self-attention mechanism.

[0089] Correspondingly, the speaker identifier can be input into the third embedding module 36 to obtain the timbre representation corresponding to the speaker identifier. Then, the timbre representation and the text encoding of the first phoneme sequence at the audio frame level are input into the decoder 35 to decode based on the timbre representation and the text encoding to obtain the predicted acoustic features.

[0090] refer to Figure 3 As can be seen, in this embodiment, the timbre representation output by the third embedding module 36 is the input of the decoder 35. The encoder 32 and the prosody prediction module 33 are unrelated to the timbre representation. Thus, only the timbre representation and the decoder 35 will affect the timbre features contained in the predicted acoustic features, thereby achieving the decoupling of text and timbre features. Furthermore, the style representation output by the second embedding module 34 is not directly added to the output of the encoder 32, but is instead stored within the prosodic prediction module 33. This is because the prosodic prediction module 33 predicts the duration, pitch, and energy features of each phoneme, which are all scalar features. When these prosodic features are added to the backbone network, it ensures that the style representation will not appear in the backbone network. The backbone network only adds prosodic features and does not contain the speaker's style representation information. Thus, even if the style representation information contains some timbre features due to the strong correlation between style features and timbre features, the prosodic prediction module can still separate the speaker's style features and predict the corresponding prosodic features using only the style features without relying on the timbre features in the style representation information. This avoids confusion between the style representation and the speaker's timbre features. Moreover, the prosodic prediction module 33's prediction of prosodic features does not depend on the timbre representation, and the timbre representation does not affect the style features, thereby achieving high-quality decoupling of timbre features and style features.

[0091] Step 207: Train the speech synthesis model based on the predicted acoustic features and sample acoustic features of each first sample audio.

[0092] The specific implementation process and principle of step 207 can be referred to the description of the above embodiments, and will not be repeated here.

[0093] Step 208: Obtain at least one sample acoustic feature of a second sample audio containing noise, the corresponding second phoneme sequence, and the corresponding speaker identifier.

[0094] Among them, the sample acoustic features of the second sample audio represent the acoustic characteristics of the second sample audio, such as Mel spectrum features, which are obtained by extracting acoustic features from the second sample audio.

[0095] The second phoneme sequence is the phoneme sequence obtained by phoneme conversion of the text corresponding to the second sample audio.

[0096] The speaker identifier corresponding to the second sample audio can be the same as or different from the speaker identifier corresponding to the first sample audio; this application does not impose any restrictions on this.

[0097] Step 209: The prosody prediction module is trained based on the sample acoustic features of at least one second sample audio containing noise, the corresponding second phoneme sequence, and the corresponding speaker identifier.

[0098] In one embodiment of this application, the first sample audio during the overall training of the speech synthesis model is clean, noise-free audio.

[0099] Furthermore, it is understandable that the extraction of pitch and energy features of phonemes is not significantly affected by noise, and Gaussian normalization can reduce the impact of noise on the features. Simultaneously, the duration features of phonemes are obtained through forced alignment and are also unaffected by noise. Therefore, the extraction of pitch, energy, and duration features of each phoneme in this embodiment does not require strictly noise-free audio. Even with noisy audio, the prosodic feature prediction module in this embodiment can extract relatively accurate prosodic features. Thus, in this embodiment, after overall training of the speech synthesis model, noisy training data can be used to train the prosodic prediction module within the speech synthesis model, allowing the prosodic prediction module to learn the style features from this training data.

[0100] For example, in this embodiment, after training the entire speech synthesis model using noise-free first sample audio, a large amount of live-streaming sales audio of a certain product can be obtained from the internet. This audio is used as a second sample audio corresponding to a speaker identifier, and the model parameters of the prosody prediction module in the speech synthesis model are updated. The model parameters of other parts of the model are not updated, thus allowing only the prosody prediction module to learn the style features in the second sample audio. After training the prosody prediction module based on the second sample audio, it is possible to synthesize a target audio with a live-streaming sales style based on the speaker identifier corresponding to any noise-free first sample audio, the speaker identifier corresponding to the live-streaming sales audio, and the target text to be synthesized. Since the prosodic features are not affected by noise, the quality of the obtained target audio is unaffected, resulting in high-quality target audio.

[0101] Therefore, by training the prosody prediction module based on second sample audio containing noise, the requirements for training data of the speech synthesis model can be reduced, and style transfer can be achieved.

[0102] Based on the training method of the speech synthesis model in the above embodiments, this application provides a speech synthesis method. The speech synthesis method provided in this application embodiment will be described below.

[0103] It should be noted that the speech synthesis method provided in this application embodiment is executed by a speech synthesis device. This speech synthesis device can be an electronic device or configured within an electronic device. By executing the speech synthesis method provided in this application embodiment, it decouples the timbre and style features in the audio, thereby flexibly generating target audio with different timbres and styles using the same speech synthesis model. This improves the flexibility of the speech synthesis model, expands its applicability, and reduces the number of models required for style transfer.

[0104] Among them, the electronic device can be a personal computer (PC), cloud device, mobile device, server, etc. The mobile device can be any hardware device such as mobile phone, tablet computer, personal digital assistant, wearable device, vehicle device, etc. This application does not limit it.

[0105] Figure 5 This is a schematic flowchart of the speech synthesis method provided in Embodiment 3 of this application. Figure 5 As shown, the speech synthesis method may include the following steps 501-504.

[0106] Step 501: Obtain the third phoneme sequence corresponding to the target text to be synthesized, and obtain the first speaker identifier and the second speaker identifier from the candidate identifier set.

[0107] The target text to be synthesized can be any text in any language type. The language type can be, for example, Chinese, English, Japanese, etc.; the target text can be, for example, news text, entertainment text, chat text, etc.; the target text to be synthesized can be text in one language type or text in multiple language types, and this disclosure does not impose any restrictions on this.

[0108] The third phoneme sequence is the phoneme sequence obtained by phoneme conversion of the target text.

[0109] The candidate identifier set consists of the speaker identifiers corresponding to each first sample audio used by the speech synthesis model during training.

[0110] The first speaker identifier is used to uniquely identify the first speaker and can be the identifier of any speaker in the candidate identifier set; the second speaker identifier is used to uniquely identify the second speaker and can be the identifier of any speaker in the candidate identifier set.

[0111] It should be noted that the data used for speech synthesis in this embodiment of the present disclosure may be provided by the user and authorized for use, or obtained from public datasets, or obtained in other ways that comply with relevant laws and regulations. This application does not impose any restrictions on this.

[0112] Step 502: Input the third phoneme sequence and the first speaker identifier into the coding layer of the speech synthesis model, so as to determine the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence based on the style representation corresponding to the first speaker identifier and the third phoneme sequence, and determine the text encoding of the third phoneme sequence at the audio frame level based on the third phoneme sequence and the prosodic features.

[0113] The style representation corresponding to the first speaker identifier is used to represent the style features of the first speaker, and can be the style embedding corresponding to the first speaker identifier.

[0114] The prosodic features of the first speaker (i.e., the first speaker) on each phoneme in the third phoneme sequence are used to characterize the first speaker's pronunciation style on each phoneme in the third phoneme sequence, which may include features such as pitch, duration, and energy.

[0115] The text encoding of the third phoneme sequence at the audio frame level includes the text encoding corresponding to multiple audio frames into which the third phoneme sequence is divided according to the audio frame level. The text encoding corresponding to each audio frame contains the features of the text corresponding to that audio frame and the prosodic features of the third speaker in that audio frame. Among them, the prosodic features of the third speaker in that audio frame are used to characterize the pronunciation style of the third speaker in that audio frame, and may include features such as pitch, duration, and energy.

[0116] In one embodiment of this application, the speech synthesis model may include an encoding layer. A first speaker identifier and a third phoneme sequence may be input into the encoding layer of the speech synthesis model. The encoding layer may query an embedding table based on the first speaker identifier to obtain the style representation corresponding to the first speaker identifier. Then, based on the style representation corresponding to the first speaker identifier and the third phoneme sequence, the encoding layer may determine the prosodic features of the first speaker on each phoneme in the third phoneme sequence, and determine the text encoding of the third phoneme sequence at the audio frame level based on the third phoneme sequence and the prosodic features.

[0117] In one embodiment of this application, the coding layer includes a first embedding module, an encoder, a prosody prediction module, and a second embedding module connected in sequence. Accordingly, step 502 can be implemented in the following way:

[0118] The first speaker identifier is input into the second embedding module to obtain the style representation corresponding to the first speaker identifier; the third phoneme sequence is input into the first embedding module to obtain the text representation of each phoneme in the third phoneme sequence, and the text representation of each phoneme is input into the encoder to obtain the text encoding of each phoneme; the text encoding and style representation of each phoneme are input into the prosody prediction module to obtain the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence, and the text encoding of the third phoneme sequence at the audio frame level is determined based on the third phoneme sequence and prosodic features.

[0119] In one embodiment of this application, the prosody prediction module includes a prosody prediction unit and a prosody processing unit; the prosody features include pitch features, energy features and duration features.

[0120] Correspondingly, the text encoding and style representation of each phoneme are input into the prosody prediction module to obtain the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence. Based on the third phoneme sequence and prosodic features, the text encoding of the third phoneme sequence at the audio frame level is determined, including: inputting the text encoding and style representation of each phoneme in the third phoneme sequence into the prosody prediction unit to obtain the pitch features, energy features, and duration features of the first speaker corresponding to the first speaker identifier on each phoneme; inputting the text encoding of each phoneme and the pitch features, energy features, and duration features of the speaker on each phoneme into the prosody processing unit to fuse the pitch features and energy features of the first speaker on each phoneme with the text encoding of each phoneme to obtain the fused encoding of each phoneme; and based on the duration features of the first speaker on each phoneme, extending the fused encoding of each phoneme to the audio frame level to obtain the text encoding of the third phoneme sequence at the audio frame level.

[0121] The process of determining the text encoding of the third phoneme sequence at the audio frame level can be referenced to the process of determining the text encoding of the first phoneme sequence at the audio frame level; the relevant descriptions will not be repeated here.

[0122] In this embodiment, the prosodic prediction module predicts prosodic features without using implicit representation encoding of prosodic features such as multi-scale prosodic representation or bottleneck features of prosodic representation (audio language features). Instead, it uses the simplest prosodic features such as pitch features, energy features, and duration features. Since style features include the duration, pitch, and energy of phonemes, and these are all scalars, the predicted prosodic features can be modified at will as needed during speech synthesis. This makes the overall speech rate of the final synthesized target audio controllable, allowing for arbitrary slowing down or speeding up of the speech rate, arbitrary adjustment of the pitch and energy of the pronunciation, and control over phoneme and phrase segmentation.

[0123] Step 503: Input the text encoding of the third phoneme sequence at the audio frame level and the second speaker identifier into the decoding layer of the speech synthesis model to decode based on the timbre representation and text encoding corresponding to the second speaker identifier to obtain acoustic features.

[0124] The speech synthesis model is trained using the training method for the speech synthesis model as described in any of the above embodiments.

[0125] The timbre representation corresponding to the second speaker identifier is used to represent the timbre features of the second speaker corresponding to the second speaker identifier, and can be the speaker embedding corresponding to the second speaker identifier.

[0126] Acoustic features, which characterize the acoustic features of the target audio, such as the Mel spectrum.

[0127] In one embodiment of this application, the speech synthesis model may include a decoding layer. After inputting the text encoding of the third phoneme sequence at the audio frame level and the corresponding second speaker identifier into the decoding layer, the decoding layer can query the embedding table based on the second speaker identifier to obtain the timbre representation corresponding to the second speaker identifier. Then, it can perform decoding based on the timbre representation and the text encoding to obtain acoustic features. These acoustic features carry text features from the target text, style features of the first speaker, and timbre features of the second speaker.

[0128] In one embodiment of this application, the decoding layer of the speech synthesis model includes a decoder connected to the encoding layer and a third embedding module connected to the decoder.

[0129] Accordingly, step 503 can be implemented as follows: input the second speaker identifier into the third embedding module to obtain the timbre representation corresponding to the second speaker identifier; input the timbre representation and the text encoding of the third phoneme sequence at the audio frame level into the decoder to decode based on the timbre representation and the text encoding to obtain acoustic features.

[0130] The process of obtaining acoustic features through the coding layer can be referenced from the process of obtaining predicted acoustic features through the coding layer during the training of a speech synthesis model; the relevant descriptions will not be repeated here.

[0131] Step 504: Based on acoustic features, generate the target audio corresponding to the target text.

[0132] In one embodiment of this application, acoustic features can be converted into speech waveforms to obtain target audio corresponding to the target text. The target audio contains style features of a first speaker and timbre features of a second speaker.

[0133] In this embodiment, since the coding layer of the speech synthesis model does not rely on timbre representation when determining the text encoding of the third phoneme sequence at the audio frame level, and timbre representation is only used by the decoding layer to determine the predicted acoustic features, only the decoding layer and timbre representation affect the timbre features carried in the acoustic features, thus achieving decoupling between text and timbre features. Furthermore, when the coding layer relies on style representation to determine the text encoding of the third phoneme sequence at the audio frame level, it does not rely on timbre representation, and the prosodic features determined based on style representation are only related to style features and not timbre features. Therefore, when the coding layer relies on style features to determine the text encoding of the third phoneme sequence at the audio frame level, it does not rely on timbre features, thus achieving decoupling between style features and timbre features.

[0134] Because the speech synthesis model decouples timbre and style features in audio, during training, the same speaker can use audio of a single style as training data. The speech synthesis model learns the individual timbre and style features of each speaker during training. Therefore, using the speech synthesis model trained in this embodiment, it can synthesize target audio with the timbre features corresponding to one speaker's identifier and the style features corresponding to the other speaker's identifier, based on any target text to be synthesized and any two speaker identifiers from the candidate identifier set. For example, during the training of the speech synthesis module, when the training data consists of the first sample audio of 10 speakers, the trained speech synthesis model can synthesize 10*10 combinations of timbre and style features. Thus, the same speech synthesis model can be used to flexibly generate target audio with different timbres and styles, improving the flexibility of the speech synthesis model, expanding its applicability, and reducing the number of models required for style transfer.

[0135] In addition, since the attention module in the backbone network of this application embodiment is not used, problems such as dropped sounds and repeated pronunciations caused by the attention mechanism will not occur, and the quality of the generated target audio is high.

[0136] For example, in this embodiment of the application, the identifiers of speaker A and speaker B, along with the target text to be synthesized, can be input into a speech synthesis model. Based on the style features of speaker A and the timbre features of speaker B, the model synthesizes the target audio 'a', which possesses the style features of speaker A and the timbre features of speaker B. Here, speaker A is male, and speaker B is female. Figure 6 The image above is a schematic diagram of the pitch features of the target audio a. Furthermore, the speaker A's identifier and the same target text to be synthesized can be input into the speech synthesis model, thereby synthesizing the target audio b, which possesses the speaker A's style and timbre features, based on the speaker A's style and timbre features. Figure 6 The following figure is a schematic diagram of the pitch characteristics of the target audio b.

[0137] refer to Figure 6 From the pitch contours, we can see that since both target audio a and target audio b are synthesized using the style features of speaker A, their pitch features are basically the same, and their styles are consistent. However, since speaker A is male and speaker B is female, then... Figure 6 In the above image, the fundamental frequency of speaker B's voice is significantly higher than... Figure 6 In the image below, speaker A has a high fundamental frequency.

[0138] Additionally, for example, if the target text is news text, in this embodiment, the identifiers of speaker B and speaker C, along with the target text to be synthesized, can be input into the speech synthesis model. This model then synthesizes the target audio c, which possesses the style features of speaker C and the timbre features of speaker B. Speakers B and C are both female. Speaker C's style is a news style. Figure 7 The image above is a schematic diagram of the pitch features of the target audio c. Furthermore, the speaker C's identifier, along with the same target text to be synthesized, can be input into the speech synthesis model. Based on the speaker C's style features and timbre features, the model synthesizes the target audio d, which possesses both the speaker C's style features and timbre features. Figure 7 The following figure is a schematic diagram of the pitch characteristics of the target audio d.

[0139] refer to Figure 7 From the pitch contour analysis, we can see that since both target audio c and target audio d are synthesized using the style features of speaker C, their pitch features are basically the same, and their styles are consistent. However, since speakers B and C are both female, then... Figure 7 The fundamental frequency of speaker B's timbre in the above image is... Figure 7 In the image below, the fundamental frequency of speaker C's timbre shows no significant difference.

[0140] Additionally, for example, if the target text is a live-streaming e-commerce text, in this embodiment, the identifiers of speaker B and speaker D, along with the target text to be synthesized, can be input into the speech synthesis model. This model then synthesizes the target audio e, which possesses both the style features of speaker D and the timbre features of speaker B. Here, both speakers B and D are female. Speaker D's style is a live-streaming e-commerce style. Figure 8 The image above is a schematic diagram of the pitch features of the target audio e. Furthermore, the speaker D's identifier, along with the same target text to be synthesized, can be input into the speech synthesis model. Based on the speaker D's style features and timbre features, the model synthesizes the target audio f, which possesses both the speaker D's style features and timbre features. Figure 8 The following figure is a schematic diagram of the pitch characteristics of the target audio f.

[0141] refer to Figure 8 From the pitch contour analysis, we can see that since both target audio e and target audio f are synthesized using the style features of speaker D, their pitch features are basically the same, and their styles are consistent. However, since speakers B and D are both female, then... Figure 8 The fundamental frequency of speaker B's timbre in the above image is... Figure 8In the image below, the fundamental frequency of speaker D's timbre shows no significant difference.

[0142] The three examples above clearly demonstrate the effectiveness of the multi-timbre and multi-style speech synthesis model based on the decoupling of timbre features and style features proposed in this application, as well as the effectiveness of the decoupling of timbre features and style features.

[0143] In summary, the speech synthesis method provided in this application obtains the third phoneme sequence corresponding to the target text to be synthesized, and obtains the first speaker identifier and the second speaker identifier from the candidate identifier set. The third phoneme sequence and the first speaker identifier are input into the encoding layer of the speech synthesis model to determine the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence based on the style representation corresponding to the first speaker identifier and the third phoneme sequence. Based on the third phoneme sequence and the prosodic features, the text encoding of the third phoneme sequence at the audio frame level is determined. The text encoding of the third phoneme sequence at the audio frame level and the second speaker identifier are input into the decoding layer of the speech synthesis model to decode based on the timbre representation corresponding to the second speaker identifier and the text encoding to obtain acoustic features. Based on the acoustic features, the target audio corresponding to the target text is generated. This achieves decoupling of timbre features and style features in the audio, thereby allowing the same speech synthesis model to flexibly generate target audio with different timbres and styles, improving the flexibility of the speech synthesis model, expanding the applicability of the speech synthesis model, and reducing the number of models required for style transfer.

[0144] Figure 9 This is a schematic diagram of the structure of the training device for the speech synthesis model provided in Embodiment 4 of this application.

[0145] like Figure 9 As shown, the training device 900 for the speech synthesis model may include: a first acquisition module 901, a first processing module 902, a second processing module 903, and a first training module 904.

[0146] The first acquisition module 901 is used to acquire the acoustic features of multiple first sample audios, the corresponding first phoneme sequences and the corresponding speaker identifiers, wherein at least one first sample audio corresponding to the same speaker identifier has a single style feature.

[0147] The first processing module 902 is used to input the first phoneme sequence corresponding to the first sample audio and the corresponding speaker identifier into the coding layer of the speech synthesis model, so as to determine the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features.

[0148] The second processing module 903 is used to input the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer of the speech synthesis model, so as to decode based on the timbre representation and text encoding corresponding to the speaker identifier to obtain the predicted acoustic features of the first sample audio.

[0149] The first training module 904 is used to train the speech synthesis model based on the predicted acoustic features and sample acoustic features of each first sample audio.

[0150] It should be noted that the speech synthesis model training device provided in this application embodiment can execute the speech synthesis model training method of the aforementioned embodiment. The speech synthesis model training device can be an electronic device or can be configured in an electronic device to achieve decoupling of timbre features and style features in audio. In this way, the speech synthesis model can be trained using audio with a single style feature corresponding to multiple speakers, thereby reducing the audio recording cost in the training data and thus reducing the training cost of the speech synthesis model.

[0151] Among them, electronic devices can be PCs, cloud devices, mobile devices, servers, etc. Mobile devices can be any hardware devices such as mobile phones, tablets, personal digital assistants, wearable devices, and in-vehicle devices, and this application does not limit them.

[0152] In one possible implementation of this application embodiment, the encoding layer includes a first embedding module, an encoder, a prosody prediction module, and a second embedding module connected in sequence; the first processing module 902 includes:

[0153] The first processing unit is used to input the speaker identifier into the second embedding module to obtain the style representation corresponding to the speaker identifier;

[0154] The second processing unit is used to input the first phoneme sequence into the first embedding module to obtain the text representation of each phoneme in the first phoneme sequence, and input the text representation of each phoneme into the encoder to obtain the text encoding of each phoneme.

[0155] The third processing unit is used to input the text encoding and style representation of each phoneme into the prosody prediction module to obtain the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence, and to determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features.

[0156] In one possible implementation of this application embodiment, the prosody prediction module includes a prosody prediction unit and a prosody processing unit; the prosodic features include pitch features, energy features, and duration features; the third processing unit is used for:

[0157] The text encoding and style representation of each phoneme are input into the prosodic prediction unit to obtain the pitch features, energy features and duration features of the speaker corresponding to the speaker identifier on each phoneme;

[0158] The text encoding of each phoneme and the speaker's pitch, energy, and duration features on each phoneme are input into the prosodic processing unit to fuse the speaker's pitch and energy features on each phoneme with the text encoding of each phoneme to obtain the fused encoding of each phoneme. Based on the speaker's duration features on each phoneme, the fused encoding of each phoneme is extended to the audio frame level to obtain the text encoding of the first phoneme sequence at the audio frame level.

[0159] In one possible implementation of this application embodiment, the decoding layer includes a decoder connected to the encoding layer and a third embedding module connected to the decoder; the second processing module 903 includes:

[0160] The fourth processing unit is used to input the speaker identifier into the third embedding module to obtain the timbre representation corresponding to the speaker identifier;

[0161] The fifth processing unit is used to input the timbre representation and the text encoding of the first phoneme sequence at the audio frame level into the decoder, so as to decode based on the timbre representation and the text encoding to obtain the predicted acoustic features.

[0162] In one possible implementation of this application embodiment, the training device 900 for the speech synthesis model further includes:

[0163] The second acquisition module is used to acquire at least one sample acoustic feature of a second sample audio containing noise, the corresponding second phoneme sequence, and the corresponding speaker identifier.

[0164] The second training module is used to train the prosody prediction module based on the sample acoustic features of at least one second sample audio containing noise, the corresponding second phoneme sequence, and the corresponding speaker identifier.

[0165] It should be noted that the explanations in the aforementioned training method embodiments of the speech synthesis model also apply to the training device of the speech synthesis model in this embodiment, and will not be repeated here.

[0166] The training apparatus for the speech synthesis model in this application embodiment acquires sample acoustic features, corresponding first phoneme sequences, and corresponding speaker identifiers for multiple first sample audios, wherein at least one first sample audio corresponding to the same speaker identifier has a single style feature; the first phoneme sequence corresponding to the first sample audio and the corresponding speaker identifier are input into the encoding layer of the speech synthesis model to determine the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and the text encoding of the first phoneme sequence at the audio frame level is determined based on the first phoneme sequence and the prosodic features; the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier are input into the decoding layer of the speech synthesis model to decode based on the timbre representation corresponding to the speaker identifier and the text encoding to obtain the predicted acoustic features of the first sample audio; the speech synthesis model is trained based on the predicted acoustic features and sample acoustic features of each first sample audio. This decouples timbre and style features in audio, allowing the speech synthesis model to be trained using audio samples from multiple speakers, each with a single style feature. This reduces the cost of recording audio data for training, and consequently, the training cost of the speech synthesis model.

[0167] Figure 10 This is a schematic diagram of the speech synthesis device provided in Embodiment 5 of this application.

[0168] like Figure 10 As shown, the speech synthesis device 1000 may include: a third acquisition module 1001, a third processing module 1002, a fourth processing module 1003, and a generation module 1004.

[0169] The third acquisition module 1001 is used to acquire the third phoneme sequence corresponding to the target text to be synthesized, and to acquire the first speaker identifier and the second speaker identifier from the candidate identifier set.

[0170] The third processing module 1002 is used to input the third phoneme sequence and the first speaker identifier into the coding layer of the speech synthesis model, so as to determine the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence based on the style representation corresponding to the first speaker identifier and the third phoneme sequence, and determine the text encoding of the third phoneme sequence at the audio frame level based on the third phoneme sequence and the prosodic features.

[0171] The fourth processing module 1003 is used to input the text encoding of the third phoneme sequence at the audio frame level and the second speaker identifier into the decoding layer of the speech synthesis model, so as to decode based on the timbre representation and text encoding corresponding to the second speaker identifier to obtain acoustic features;

[0172] The generation module 1004 is used to generate target audio corresponding to the target text based on acoustic features.

[0173] It should be noted that the speech synthesis device provided in this application embodiment can execute the speech synthesis method of the foregoing embodiment. The speech synthesis device can be an electronic device or can be configured in an electronic device to achieve decoupling of timbre features and style features in audio, thereby using the same speech synthesis model to flexibly generate target audio with different timbres and styles, improve the flexibility of the speech synthesis model, expand the applicability of the speech synthesis model, and reduce the number of models required for style transfer.

[0174] Among them, electronic devices can be PCs, cloud devices, mobile devices, servers, etc. Mobile devices can be any hardware devices such as mobile phones, tablets, personal digital assistants, wearable devices, and in-vehicle devices, and this application does not limit them.

[0175] It should be noted that the explanations and descriptions in the aforementioned speech synthesis method embodiments also apply to the speech synthesis device of this embodiment, and will not be repeated here.

[0176] The speech synthesis apparatus of this application embodiment obtains the third phoneme sequence corresponding to the target text to be synthesized, and obtains the first speaker identifier and the second speaker identifier from the candidate identifier set. The third phoneme sequence and the first speaker identifier are input into the encoding layer of the speech synthesis model to determine the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence based on the style representation corresponding to the first speaker identifier and the third phoneme sequence. The text encoding of the third phoneme sequence at the audio frame level is determined based on the third phoneme sequence and the prosodic features. The text encoding of the third phoneme sequence at the audio frame level and the second speaker identifier are input into the decoding layer of the speech synthesis model to decode based on the timbre representation corresponding to the second speaker identifier and the text encoding to obtain acoustic features. Based on the acoustic features, the target audio corresponding to the target text is generated. This achieves decoupling of timbre features and style features in the audio, thereby allowing the same speech synthesis model to flexibly generate target audio with different timbres and styles, improving the flexibility of the speech synthesis model, expanding the applicability of the speech synthesis model, and reducing the number of models required for style transfer.

[0177] To implement the above embodiments, this application also proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a training method for a speech synthesis model as proposed in any of the foregoing embodiments of this application, or to perform a speech synthesis method as proposed in any of the foregoing embodiments of this application.

[0178] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute a training method for a speech synthesis model as proposed in any of the foregoing embodiments of this application, or to execute a speech synthesis method as proposed in any of the foregoing embodiments of this application.

[0179] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements a training method for a speech synthesis model as proposed in any of the foregoing embodiments of this application, or implements a speech synthesis method as proposed in any of the foregoing embodiments of this application.

[0180] Figure 11 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0181] like Figure 11 As shown, the electronic device 1100 is presented in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0182] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0183] Electronic device 1100 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 1100, including volatile and non-volatile media, removable and non-removable media.

[0184] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 1100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 11 Not shown; usually referred to as a "hard drive"). Although Figure 11 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0185] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this application.

[0186] Electronic device 1100 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 1100, and / or with any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 1100 via bus 18. It should be understood that, although... Figure 11As not shown, other hardware and / or software modules may be used in conjunction with electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0187] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the foregoing embodiments.

[0188] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0189] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0190] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0191] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0192] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0193] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0195] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A training method for a speech synthesis model, characterized in that, The method includes: Acquire the acoustic features of multiple first sample audios, the corresponding first phoneme sequences, and the corresponding speaker identifiers, wherein at least one first sample audio corresponding to the same speaker identifier has a single style feature; The first phoneme sequence corresponding to the first sample audio and the corresponding speaker identifier are input into the encoding layer of the speech synthesis model. Based on the style representation corresponding to the speaker identifier and the first phoneme sequence, the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence are determined. Based on the first phoneme sequence and the prosodic features, the text encoding of the first phoneme sequence at the audio frame level is determined. The text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier are input into the decoding layer of the speech synthesis model to decode based on the timbre representation corresponding to the speaker identifier and the text encoding, so as to obtain the predicted acoustic features of the first sample audio. The speech synthesis model is trained based on the predicted acoustic features of each of the first sample audio samples and the acoustic features of the samples.

2. The method according to claim 1, characterized in that, The coding layer includes a first embedding module, an encoder, a prosody prediction module, and a second embedding module connected in sequence. The step of inputting the first phoneme sequence corresponding to the first sample audio and the corresponding speaker identifier into the encoding layer of the speech synthesis model, so as to determine the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and to determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features, includes: The speaker identifier is input into the second embedding module to obtain the style representation corresponding to the speaker identifier; The first phoneme sequence is input into the first embedding module to obtain the text representation of each phoneme in the first phoneme sequence, and the text representation of each phoneme is input into the encoder to obtain the text encoding of each phoneme. The text encoding of each phoneme and the style representation are input into the prosodic prediction module to obtain the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence, and the text encoding of the first phoneme sequence at the audio frame level is determined based on the first phoneme sequence and the prosodic features.

3. The method according to claim 2, characterized in that, The prosodic prediction module includes a prosodic prediction unit and a prosodic processing unit; the prosodic features include pitch features, energy features, and duration features. The step of inputting the text encoding of each phoneme and the style representation into the prosodic prediction module to obtain the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence, and determining the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features, includes: The text encoding and style representation of each phoneme are input into the prosodic prediction unit to obtain the pitch features, energy features and duration features of the speaker corresponding to the speaker identifier on each phoneme; The text encoding of each phoneme and the pitch, energy, and duration features of the speaker on each phoneme are input into the prosody processing unit to fuse the pitch and energy features of the speaker on each phoneme with the text encoding of each phoneme to obtain the fused encoding of each phoneme. Based on the duration features of the speaker on each phoneme, the fused encoding of each phoneme is extended to the audio frame level to obtain the text encoding of the first phoneme sequence at the audio frame level.

4. The method according to any one of claims 1-3, characterized in that, The decoding layer includes a decoder connected to the encoding layer and a third embedding module connected to the decoder; The step of inputting the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer of the speech synthesis model, and decoding based on the timbre representation corresponding to the speaker identifier and the text encoding to obtain the predicted acoustic features of the first sample audio, includes: The speaker identifier is input into the third embedding module to obtain the timbre representation corresponding to the speaker identifier; The timbre representation and the text encoding of the first phoneme sequence at the audio frame level are input into the decoder to perform decoding based on the timbre representation and the text encoding to obtain the predicted acoustic features.

5. The method according to claim 2 or 3, characterized in that, After training the speech synthesis model based on the predicted acoustic features of each of the first sample audio samples and the sample acoustic features, the method further includes: Acquire at least one sample acoustic feature of a second sample audio containing noise, the corresponding second phoneme sequence, and the corresponding speaker identifier; The prosody prediction module is trained based on the sample acoustic features of at least one noisy second sample audio, the corresponding second phoneme sequence, and the corresponding speaker identifier.

6. A speech synthesis method, characterized in that, The method includes: Obtain the third phoneme sequence corresponding to the target text to be synthesized, and obtain the first speaker identifier and the second speaker identifier from the candidate identifier set; The third phoneme sequence and the first speaker identifier are input into the encoding layer of the speech synthesis model to determine the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence based on the style representation corresponding to the first speaker identifier and the third phoneme sequence, and to determine the text encoding of the third phoneme sequence at the audio frame level based on the third phoneme sequence and the prosodic features, wherein the speech synthesis model is obtained by the training method according to any one of claims 1-5; The text encoding of the third phoneme sequence at the audio frame level and the second speaker identifier are input into the decoding layer of the speech synthesis model to decode based on the timbre representation corresponding to the second speaker identifier and the text encoding to obtain acoustic features; Based on the acoustic features, the target audio corresponding to the target text is generated.

7. A training device for a speech synthesis model, characterized in that, The device includes: The first acquisition module is used to acquire the acoustic features of multiple first sample audios, the corresponding first phoneme sequences and the corresponding speaker identifiers, wherein at least one first sample audio corresponding to the same speaker identifier has a single style feature. The first processing module is used to input the first phoneme sequence corresponding to the first sample audio and the corresponding speaker identifier into the encoding layer of the speech synthesis model, so as to determine the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence based on the style representation corresponding to the speaker identifier and the first phoneme sequence, and determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features. The second processing module is used to input the text encoding of the first phoneme sequence at the audio frame level and the corresponding speaker identifier into the decoding layer of the speech synthesis model, so as to decode based on the timbre representation corresponding to the speaker identifier and the text encoding to obtain the predicted acoustic features of the first sample audio. The first training module is used to train the speech synthesis model based on the predicted acoustic features of each of the first sample audios and the sample acoustic features.

8. The apparatus according to claim 7, characterized in that, The coding layer includes a first embedding module, an encoder, a prosody prediction module, and a second embedding module connected in sequence. The first processing module includes: The first processing unit is used to input the speaker identifier into the second embedding module to obtain the style representation corresponding to the speaker identifier; The second processing unit is used to input the first phoneme sequence into the first embedding module to obtain the text representation of each phoneme in the first phoneme sequence, and input the text representation of each phoneme into the encoder to obtain the text encoding of each phoneme. The third processing unit is used to input the text encoding of each phoneme and the style representation into the prosody prediction module to obtain the prosodic features of the speaker corresponding to the speaker identifier on each phoneme in the first phoneme sequence, and to determine the text encoding of the first phoneme sequence at the audio frame level based on the first phoneme sequence and the prosodic features.

9. The apparatus according to claim 8, characterized in that, The prosodic prediction module includes a prosodic prediction unit and a prosodic processing unit; the prosodic features include pitch features, energy features, and duration features; the third processing unit is used for: The text encoding and style representation of each phoneme are input into the prosodic prediction unit to obtain the pitch features, energy features and duration features of the speaker corresponding to the speaker identifier on each phoneme; The text encoding of each phoneme and the pitch, energy, and duration features of the speaker on each phoneme are input into the prosody processing unit to fuse the pitch and energy features of the speaker on each phoneme with the text encoding of each phoneme to obtain the fused encoding of each phoneme. Based on the duration features of the speaker on each phoneme, the fused encoding of each phoneme is extended to the audio frame level to obtain the text encoding of the first phoneme sequence at the audio frame level.

10. The apparatus according to any one of claims 7-9, characterized in that, The decoding layer includes a decoder connected to the encoding layer and a third embedding module connected to the decoder; The second processing module includes: The fourth processing unit is used to input the speaker identifier into the third embedding module to obtain the timbre representation corresponding to the speaker identifier; The fifth processing unit is used to input the timbre representation and the text encoding of the first phoneme sequence at the audio frame level into the decoder, so as to decode based on the timbre representation and the text encoding to obtain the predicted acoustic features.

11. The apparatus according to claim 8 or 9, characterized in that, The device further includes: The second acquisition module is used to acquire at least one sample acoustic feature of a second sample audio containing noise, the corresponding second phoneme sequence, and the corresponding speaker identifier. The second training module is used to train the prosody prediction module based on the sample acoustic features of the at least one second sample audio containing noise, the corresponding second phoneme sequence, and the corresponding speaker identifier.

12. A speech synthesis device, characterized in that, The device includes: The third acquisition module is used to acquire the third phoneme sequence corresponding to the target text to be synthesized, and to acquire the first speaker identifier and the second speaker identifier from the candidate identifier set; The third processing module is used to input the third phoneme sequence and the first speaker identifier into the encoding layer of the speech synthesis model, so as to determine the prosodic features of the speaker corresponding to the first speaker identifier on each phoneme in the third phoneme sequence based on the style representation corresponding to the first speaker identifier and the third phoneme sequence, and determine the text encoding of the third phoneme sequence at the audio frame level based on the third phoneme sequence and the prosodic features, wherein the speech synthesis model is obtained by the training method according to any one of claims 1-5; The fourth processing module is used to input the text encoding of the third phoneme sequence at the audio frame level and the second speaker identifier into the decoding layer of the speech synthesis model, so as to decode based on the timbre representation corresponding to the second speaker identifier and the text encoding to obtain acoustic features; The generation module is used to generate target audio corresponding to the target text based on the acoustic features.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5, or to perform the method of claim 6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method of any one of claims 1-5, or to perform the method of claim 6.

Images