Speech synthesis methods, devices, electronic devices and readable storage media

Abstract

This disclosure relates to a speech synthesis method, apparatus, electronic device, and readable storage medium. The method analyzes the text to be processed based on a speech synthesis model and outputs the spectral features corresponding to the text. The speech synthesis model includes a prosody sub-model and a timbre sub-model. The prosody sub-model receives the text to be processed as input and outputs a first acoustic feature corresponding to the text, including bottleneck features characterizing the target rap style. The timbre sub-model receives the first acoustic feature as input and outputs corresponding spectral features, including spectral features characterizing the target timbre. The spectral features are then transformed to obtain audio with the target rap style and target timbre, meeting the user's personalized needs for synthesized audio. The speech synthesis model supports the transformation of any text to be processed, reducing the requirements for the user's music creation ability and encouraging the user's enthusiasm for creating multimedia content.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the continuous development of internet technology, applications can now support users in creating creative videos. When creating creative videos, it's common to add background music. Currently, adding background music usually involves selecting music from a music library, which fails to meet users' personalized needs. Summary of the Invention

[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a speech synthesis method, apparatus, electronic device and readable storage medium.

[0004] In a first aspect, this disclosure provides a speech synthesis method, including:

[0005] Get the text to be processed;

[0006] The text to be processed is input into a speech synthesis model, and the spectral features corresponding to the text to be processed are obtained from the output of the speech synthesis model. The speech synthesis model includes a prosody sub-model and a timbre sub-model. The prosody sub-model outputs a first acoustic feature corresponding to the input text to be processed, the first acoustic feature including bottleneck features characterizing the target rap style. The timbre sub-model outputs spectral features corresponding to the input first acoustic feature, the spectral features including spectral features characterizing the target timbre.

[0007] Based on the spectral features corresponding to the text to be processed, the target audio corresponding to the text to be processed is obtained, and the target audio has the target timbre and the target rap style.

[0008] In some possible implementations, the prosodic sub-model is obtained by training based on the labeled text corresponding to the first sample audio and the second acoustic features corresponding to the first sample audio;

[0009] The first sample audio includes at least one audio of the target rap style; the second acoustic feature includes the first labeled bottleneck feature corresponding to the first sample audio.

[0010] In some possible implementations, the timbre sub-model is obtained by training based on the third acoustic feature corresponding to the second sample audio, the first labeled spectral feature corresponding to the second sample audio, the fourth acoustic feature corresponding to the third sample audio, and the second labeled spectral feature corresponding to the third sample audio.

[0011] The third acoustic feature includes the second labeled bottleneck feature corresponding to the second sample audio; the third sample audio includes at least one audio with the target timbre; and the fourth acoustic feature corresponding to the third sample audio includes the third labeled bottleneck feature corresponding to the third sample audio.

[0012] In some possible implementations, the first annotation bottleneck feature corresponding to the first sample audio, the second annotation bottleneck feature corresponding to the second sample audio, and the third annotation bottleneck feature corresponding to the third sample audio are obtained by the encoder of the end-to-end speech recognition model extracting bottleneck features from the input first sample audio, second sample audio, and third sample audio, respectively.

[0013] In some possible implementations, the second acoustic feature further includes: a first labeled fundamental frequency feature corresponding to the first sample audio;

[0014] The third acoustic feature further includes: the second labeled fundamental frequency feature corresponding to the second sample audio; the fourth acoustic feature further includes: the third labeled fundamental frequency feature corresponding to the third sample audio;

[0015] The first acoustic feature also includes: the fundamental frequency feature corresponding to the text to be processed.

[0016] In some possible implementations, the method further includes:

[0017] Add the target audio corresponding to the text to be processed to the target multimedia content.

[0018] Secondly, this disclosure provides a speech synthesis apparatus, comprising:

[0019] The acquisition module is used to acquire the text to be processed;

[0020] A processing module is used to input the text to be processed into a speech synthesis model and obtain the spectral features corresponding to the text to be processed output by the speech synthesis model; wherein, the speech synthesis model includes a prosody sub-model and a timbre sub-model, the prosody sub-model is used to output a first acoustic feature corresponding to the text to be processed based on the input text to be processed, the first acoustic feature including bottleneck features for characterizing the target rap style; the timbre sub-model is used to input the spectral features corresponding to the text to be processed based on the input first acoustic feature, the spectral features corresponding to the text to be processed including spectral features for characterizing the target timbre;

[0021] The processing module is used to obtain the target audio corresponding to the text to be processed based on the spectral features of the text to be processed, wherein the target audio has the target timbre and the target rap style.

[0022] Thirdly, this disclosure provides an electronic device, including: a memory, a processor, and a computer program;

[0023] The memory is configured to store the computer program;

[0024] The processor is configured to execute the computer program to implement the speech synthesis method as described in any of the first aspects.

[0025] Fourthly, this disclosure provides a readable storage medium, including: a computer program;

[0026] When the computer program is executed by at least one processor of an electronic device, it implements the speech synthesis method as described in any of the first aspects.

[0027] Fifthly, this disclosure provides a program product comprising: a computer program; the computer program is stored in a readable storage medium, an electronic device retrieves the computer program from the readable storage medium, and at least one processor of the electronic device, when processing the computer program, implements the speech synthesis method as described in any of the first aspects.

[0028] This disclosure provides a speech synthesis method, apparatus, electronic device, and readable storage medium. The method analyzes the text to be processed based on a speech synthesis model and outputs the spectral features corresponding to the text. The speech synthesis model includes a prosody sub-model and a timbre sub-model. The prosody sub-model receives the text to be processed as input and outputs a first acoustic feature corresponding to the text, which includes bottleneck features characterizing the target rap style. The timbre sub-model receives the first acoustic feature as input and outputs the spectral features corresponding to the text, including spectral features characterizing the target timbre. By converting the spectral features output by the speech synthesis model, rap audio with the target rap style and timbre can be obtained, satisfying users' personalized needs for synthesized audio. Furthermore, the speech synthesis model supports the conversion of any text to be processed, reducing the requirements for users' music creation abilities and encouraging users to create multimedia content. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0030] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figures 1a to 1c This is a schematic diagram of the structure of a speech synthesis model provided in an embodiment of the present disclosure;

[0032] Figure 2 A flowchart of a speech synthesis method provided in an embodiment of this disclosure;

[0033] Figure 3 A flowchart of a speech synthesis method provided in another embodiment of this disclosure;

[0034] Figure 4 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present disclosure;

[0035] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0036] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0037] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0038] This disclosure provides a speech synthesis method, apparatus, electronic device, readable storage medium, and program product. The method uses a pre-trained speech synthesis model to convert text into audio with a target rap style and target timbre. The speech synthesis model can achieve relatively independent control of speech synthesis based on the target rap style and timbre, thereby meeting the user's needs for personalized speech synthesis.

[0039] The target rap style mentioned in this disclosure can include any category of rap style, and this disclosure does not limit the specific type of rap style that the target rap style can be. For example, the target rap style can be any of the following rap styles: pop rap, alternative rap, comedy rap, jazz rap, and hip hop rap.

[0040] The speech synthesis method provided in this disclosure can be executed by an electronic device. This electronic device can be a tablet computer, mobile phone (such as a foldable phone, a large-screen phone, etc.), wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), smart TV, smart screen, high-definition TV, 4K TV, smart speaker, smart projector, and other Internet of Things (IoT) devices. This disclosure does not impose any restrictions on the specific type of electronic device.

[0041] It should be noted that the electronic device used to train and acquire the speech synthesis model and the electronic device used to execute speech synthesis services using the speech synthesis model can be different electronic devices or the same electronic device; this disclosure does not limit this. For example, the speech synthesis model can be trained and acquired by a server device, and then the server device can distribute the trained speech synthesis model to the terminal device / server device, which can then execute speech synthesis services based on the speech synthesis model. Alternatively, the speech synthesis model can be trained and acquired by a server device, and then the trained speech synthesis model can be deployed on the server device, which can then call the speech synthesis model to process speech synthesis services. This disclosure does not impose any restrictions, and in practical applications, it can be flexibly configured.

[0042] The speech synthesis model in this scheme will be introduced below.

[0043] The speech synthesis model in this scheme decouples itself into two sub-models by introducing acoustic features, including bottleneck features: a prosodic sub-model and a timbre sub-model. The prosodic sub-model is used to establish a deep mapping between text and acoustic features containing bottleneck features; the timbre sub-model is used to establish a deep mapping between acoustic features containing bottleneck features and spectral features.

[0044] Based on this, at least the following beneficial effects are achieved:

[0045] 1. The two decoupled feature extraction sub-models can be trained using different sample audio.

[0046] The prosodic sub-model is used to establish a deep mapping between text sequences and acoustic features containing bottleneck features. The prosodic sub-model requires high-quality first sample audio with the target rap style and the corresponding labeled text as sample data to train the prosodic sub-model.

[0047] The timbre sub-model is used to establish a deep mapping between acoustic features containing bottleneck features and spectral features. The timbre sub-model can be trained using second sample audio without corresponding text annotation. Since there is no need to annotate the text corresponding to the second sample audio, this can greatly reduce the cost of obtaining the second sample audio.

[0048] 2. By decoupling the speech synthesis model, the speech synthesis can be controlled relatively independently by the rap style and timbre.

[0049] The acoustic features output by the prosodic sub-model include bottleneck features to characterize the target rap style, enabling control of speech synthesis by the rap style. Additionally, the acoustic features output by the prosodic sub-model may also include fundamental frequency features to characterize pitch, enabling control of pitch by the speech synthesis.

[0050] The spectral features of the text output by the timbre sub-model include spectral features used to characterize the target timbre, thereby enabling the control of timbre over speech synthesis.

[0051] Furthermore, it should be noted that the spectral features output by the timbre sub-model also include spectral features characterizing the target rap style, and the spectral features characterizing the target timbre and the spectral features characterizing the target rap style are the same. If the acoustic features output by the prosody sub-model also include fundamental frequency features, then the spectral features output by the timbre sub-model also include spectral features characterizing the corresponding fundamental frequency, and the spectral features characterizing the target timbre, the spectral features characterizing the target rap style, and the spectral features characterizing the fundamental frequency are the same.

[0052] 3. Reduced the requirements for third-sample audio with the target timbre.

[0053] This speech synthesis model can be trained with a small number of third-sample audio samples of the target timbre, and the final speech synthesis model can synthesize audio with the target timbre. Even if the quality of the third-sample audio is not high, such as non-standard pronunciation or disfluency, the speech synthesis model can still stably synthesize audio with the target timbre.

[0054] Since the timbre sub-model has been trained using the second sample audio, it already possesses a high level of speech synthesis control capability for timbre. Therefore, even if the timbre sub-model learns a small amount of third sample audio, it can still master the target timbre quite well.

[0055] The following detailed description, through several specific embodiments, illustrates the structure of a speech synthesis model and how to train and obtain it. In these embodiments, an electronic device is used as an example, and the accompanying drawings are provided for detailed explanation.

[0056] in, Figure 1a This diagram illustrates the overall framework for training and acquiring a speech synthesis model. Figure 1b and Figure 1c The structural diagrams of the prosody sub-model and timbre sub-model included in the speech synthesis model are shown as examples.

[0057] Reference Figure 1a As shown, the speech synthesis model 100 includes a prosody sub-model 101 and a timbre sub-model 102. The training process for the speech synthesis model 100 includes training the prosody sub-model 101 and training the timbre sub-model 102.

[0058] The following sections describe the training process for the prosody sub-model 101 and the timbre sub-model 102, respectively.

[0059] I. Training the prosody sub-model 101

[0060] The prosodic sub-model 101 is trained based on the labeled text corresponding to the first sample audio and the labeled acoustic features (hereinafter referred to as the labeled acoustic features corresponding to the first sample audio as the second acoustic features). By learning the relationship between the labeled text corresponding to the first sample audio and the second acoustic features, the prosodic sub-model 101 acquires the ability to establish a deep mapping between the text and the acoustic features containing the bottleneck features.

[0061] Specifically, the prosody sub-model 101 is used to analyze the labeled text corresponding to the first sample audio input, model the intermediate feature sequence, perform feature transformation and dimensionality reduction on the intermediate feature sequence, and output the fifth acoustic feature corresponding to the labeled text.

[0062] Then, based on the second acoustic feature corresponding to the first sample audio, the fifth acoustic feature corresponding to the first sample audio, and the pre-constructed loss function, the loss function information for this round of training is calculated, and the coefficient values ​​of the parameters included in the prosodic sub-model 101 are adjusted according to the loss function information for this round of training.

[0063] Through iterative training of multiple first sample audios, the labeled text corresponding to the first sample audios, and the second acoustic features (including the first labeled bottleneck features) corresponding to the first sample audios, a first feature extraction model 101 that satisfies the corresponding convergence conditions is finally obtained.

[0064] During the training process, the second acoustic feature corresponding to the first sample audio can be understood as the learning target of the prosodic sub-model 101.

[0065] The first sample audio can include high-quality audio files (high-quality audio text can also be understood as clean audio), and the annotation text corresponding to the first sample audio can include one or more characters or phonemes corresponding to the first sample audio; this disclosure does not limit this. The first sample audio can be obtained by recording and cleaning multiple times according to actual needs, or it can be obtained by filtering and cleaning multiple times from an audio database; this disclosure does not restrict the method of obtaining the first sample audio. Similarly, the annotation text corresponding to the first sample audio can also be obtained through repeated annotation and correction to ensure the accuracy of the annotation text.

[0066] Furthermore, the first sample audio mentioned in this disclosure is audio with the target rap style. This disclosure does not limit the parameters such as the duration, file format, and number of the first sample audio, and the first sample audio can be a music segment sung by the same or different singers.

[0067] Furthermore, the fifth acoustic feature corresponding to the annotated text can be understood as the predicted acoustic feature corresponding to the annotated text output by the prosodic sub-model 101, and the fifth acoustic feature corresponding to the annotated text can also be understood as the fifth acoustic feature corresponding to the first sample audio.

[0068] In some embodiments, the second acoustic feature includes: a first labeled bottleneck feature corresponding to the first sample audio.

[0069] Among them, the bottleneck is a non-linear feature transformation technique and an effective dimensionality reduction technique. In the speech synthesis scenario for specific timbres mentioned in this solution, the bottleneck feature can include information in dimensions such as prosody and content.

[0070] One possible implementation is that the first labeled bottleneck feature corresponding to the first sample audio can be obtained by the encoder of the end-to-end speech recognition (ASR) model.

[0071] In the following text, the end-to-end ASR model will be referred to as the ASR model.

[0072] For example, refer to Figure 1a As shown, the first sample audio can be input into the ASR model 104 to obtain the first labeled bottleneck feature corresponding to the first sample audio output by the encoder of the ASR model 104. The encoder of the ASR model 104 is equivalent to the extractor of the bottleneck feature in advance. In this scheme, the encoder of the ASR model 104 can be used to prepare sample data.

[0073] It should be noted that ASR model 104 may also include other modules, such as Figure 1a As shown, the ASR model 104 also includes a decoder and an attention network. No processing is required on the outputs of modules other than the encoder in the ASR model 104, and this disclosure does not limit the functionality or implementation of modules or networks other than the encoder in the ASR model.

[0074] The example of obtaining the first labeled bottleneck feature corresponding to the first sample audio through the encoder of ASR model 104 is merely an example and not a limitation on the implementation method of obtaining the first labeled bottleneck feature corresponding to the first sample audio. In practical applications, it can also be obtained through other methods, and this disclosure does not impose any restrictions on this. For example, the first sample audio and the first labeled bottleneck feature corresponding to the first sample audio can be stored in a database, and the electronic device can also obtain the first sample audio and the first labeled bottleneck feature from the database.

[0075] In other embodiments, the second acoustic feature corresponding to the first sample audio includes: a first labeled bottleneck feature corresponding to the first sample audio and a first labeled fundamental frequency feature corresponding to the first sample audio.

[0076] The first labeled bottleneck feature can be described in detail in the previous example; for the sake of brevity, it will not be repeated here.

[0077] Pitch represents the subjective perception of the highness or lowness of a sound by the human ear. The pitch primarily depends on the fundamental frequency (FRF) of the sound; a higher FRF results in a higher pitch, and a lower FRF results in a lower pitch. In speech synthesis, pitch is also a crucial factor affecting the synthesis effect. To enable the final speech synthesis model to control pitch, this scheme introduces both bottleneck features and fundamental frequency features, allowing the final prosodic sub-model 101 to output corresponding bottleneck and fundamental frequency features based on the input text.

[0078] Specifically, the prosody sub-model 101 is used to analyze the labeled text corresponding to the first sample audio input, model the intermediate feature sequence, perform feature transformation and dimensionality reduction on the intermediate feature sequence, and output the fifth acoustic feature corresponding to the labeled text.

[0079] The fifth acoustic feature corresponding to the annotated text can be understood as the predicted acoustic feature corresponding to the annotated text output by the prosodic sub-model 101. The fifth acoustic feature corresponding to the annotated text can also be understood as the fifth acoustic feature corresponding to the first sample audio.

[0080] It should be noted that the second acoustic feature corresponding to the first sample audio includes the first labeled bottleneck feature and the first labeled fundamental frequency feature. Therefore, during the training process, the fifth acoustic feature corresponding to the first sample audio output by the prosody sub-model 101 also includes the prediction bottleneck feature and the prediction fundamental frequency feature corresponding to the first sample audio.

[0081] Then, based on the second acoustic feature corresponding to the first sample audio, the fifth acoustic feature corresponding to the first sample audio, and the pre-constructed loss function, the loss function information for this round of training is calculated, and the coefficient values ​​of the parameters included in the prosodic sub-model 101 are adjusted according to the loss function information.

[0082] Through iterative training of massive amounts of first sample audio, the labeled text corresponding to the first sample audio, and the second acoustic features (including the first labeled bottleneck feature and the first labeled fundamental frequency feature) corresponding to the first sample audio, a first feature extraction model 101 that satisfies the corresponding convergence condition is finally obtained.

[0083] During the training process, the second acoustic feature corresponding to the first sample audio can be understood as the learning target of the prosodic sub-model 101.

[0084] One possible implementation is that the first labeled fundamental frequency feature corresponding to the first sample audio can be obtained by analyzing the first sample audio using digital signal processing (DSP) methods. For example, such as... Figure 1aAs shown, the first sample audio can be digitally processed by the digital signal processor 105 to obtain the first labeled fundamental frequency feature corresponding to the first sample audio. The specific implementation of the digital signal processor 105 is not limited, as long as it can extract the first labeled fundamental frequency feature corresponding to the input first sample audio.

[0085] Furthermore, the first labeled fundamental frequency feature corresponding to the first sample audio is not limited to being obtained through digital signal processing methods, and this disclosure does not limit the implementation method for obtaining the first labeled fundamental frequency feature. For example, some databases store the first sample audio and the first labeled fundamental frequency feature corresponding to the first sample audio, and the first sample audio and the first labeled fundamental frequency feature can also be obtained from the database.

[0086] It should be noted that the convergence conditions for the prosodic sub-model may include, but are not limited to, evaluation metrics such as the number of iterations and the loss threshold. This disclosure does not impose restrictions on the convergence conditions for training the prosodic sub-model. Furthermore, the electronic device may be trained based on the first labeled bottleneck features corresponding to the first sample audio, or it may be trained based on both the first labeled bottleneck features and the first labeled fundamental frequency features corresponding to the first sample audio; the convergence conditions may differ.

[0087] Furthermore, the electronic device is trained based on the first labeled bottleneck features corresponding to the first sample audio, or it is trained based on the first labeled bottleneck features and the first labeled fundamental frequency features corresponding to the first sample audio. The loss functions corresponding to the pre-constructed prosodic sub-models can be the same or different. This disclosure does not limit the implementation method of the loss function corresponding to the pre-constructed prosodic sub-model.

[0088] The following is an example of the network structure of the prosody sub-model.

[0089] Figure 1b An exemplary implementation of the prosody sub-model 101 is shown. (Refer to...) Figure 1b As shown, the prosody sub-model 101 may include: a text encoder network 1011, an attention network 1012, and a decoder network 1013.

[0090] The text encoding network 1011 is used to receive text as input, analyze the context and temporal relationship of the input text, and model the intermediate feature sequence, which contains contextual information and temporal relationship.

[0091] Decoding network 1013 can adopt an autoregressive network structure, using the output of the previous time step as the input of the next time step.

[0092] Attention network 1012 is primarily used to output attention coefficients. These attention coefficients are then weighted and averaged with the intermediate feature sequence output by text encoding network 1011 to obtain a weighted average result. This weighted average result serves as another conditional input to decoding network 1013 at each time step. Decoding network 1013 performs feature transformation on the input (i.e., the weighted average result and the output of the previous time step) to output the predicted acoustic features corresponding to the text.

[0093] Combining the two aforementioned implementation methods, the predicted acoustic features corresponding to the text output by the decoding network 1013 may include: the predicted bottleneck features corresponding to the text; or, the predicted acoustic features corresponding to the text output by the decoding network 1013 may include: the predicted bottleneck features corresponding to the text and the predicted fundamental frequency features corresponding to the text.

[0094] Furthermore, the initial values ​​of the coefficients of the parameters included in the prosody sub-model 101 may be randomly generated, preset, or determined in other ways, and this disclosure does not limit this.

[0095] The prosodic sub-model 101 is iteratively trained using the labeled text corresponding to multiple first sample audios and the second acoustic features corresponding to the first sample audios. The coefficient values ​​of the parameters included in the prosodic sub-model 101 are continuously optimized until the convergence condition of the prosodic sub-model 101 is met, at which point the training of the prosodic sub-model 101 is stopped.

[0096] It should be understood that the first sample audio described above corresponds one-to-one with the corresponding labeled text, and they are paired sample data.

[0097] II. Training the timbre sub-model 102

[0098] Training the timbre sub-model 102 involves two stages. The first stage trains the timbre sub-model based on the second sample audio to obtain an intermediate model. The second stage fine-tunes the intermediate model based on the third sample audio to obtain the final timbre sub-model.

[0099] In this disclosure, the timbre of the second sample audio is not limited; in addition, the third sample audio is a sample audio with the target timbre.

[0100] It should be noted that the spectral features output by the aforementioned timbre sub-model can be Mel spectral features, or other types of spectral features. In the following example, we will use the first labeled spectral feature corresponding to the second sample audio input to the timbre sub-model as the first labeled Mel spectral feature, the second labeled spectral feature corresponding to the third sample audio as the second labeled Mel spectral feature, and the predicted spectral feature output by the timbre sub-model as the predicted Mel spectral feature as an example for illustration.

[0101] The training process of timbre sub-model 102 is described in detail below:

[0102] Phase 1:

[0103] In the first stage of training, the timbre sub-model 102 is used for iterative training based on the second sample audio to obtain an intermediate model.

[0104] The timbre sub-model 102 obtains an intermediate model with certain speech synthesis control capabilities for timbre by learning the mapping relationship between the third acoustic feature corresponding to the second sample audio and the first labeled Mel spectral feature of the second sample audio. The first labeled Mel spectral feature includes: spectral features used to characterize the timbre of the corresponding second sample audio.

[0105] This disclosure does not limit parameters such as the timbre, duration, storage format, or number of second sample audio files. The second sample audio files may include audio with a specific target timbre, audio with a non-target timbre, or both target and non-target timbre audio files.

[0106] In the first stage of training, the timbre sub-model 102 is used to analyze the second acoustic features corresponding to the input second sample audio and output the predicted Mel spectrum features corresponding to the second sample audio. Then, based on the first labeled Mel spectrum features corresponding to the second sample audio and the predicted Mel spectrum features corresponding to the second sample audio, the coefficient values ​​of the parameters included in the timbre sub-model 102 are adjusted. Through continuous iterative training of the timbre sub-model 102 with a large number of second sample audio samples, an intermediate model is obtained.

[0107] During the first phase of training, the first labeled Mel spectral features can be understood as the learning objective of the timbre sub-model 102 in the first phase.

[0108] Since the input to the timbre sub-model 102 is the third acoustic feature corresponding to the second sample audio, the second sample audio does not need to be labeled with corresponding text, thus greatly reducing the time and manpower costs associated with acquiring the second sample audio. Furthermore, a large amount of audio can be obtained at a low cost as second sample audio for iterative training of the timbre sub-model 102. By training the timbre sub-model 102 with a large amount of second sample audio, the intermediate model acquires a high level of control over timbre-specific speech synthesis.

[0109] Phase Two:

[0110] In the second stage, the intermediate model is trained based on the third sample, so that the intermediate model learns the target timbre and obtains the speech synthesis control capability for the target timbre.

[0111] It should be noted that since the intermediate model already has a high degree of speech synthesis control capability for timbre, the requirements for the third sample audio have been reduced. For example, the requirements for the duration and quality of the third sample audio have been reduced. Even if the duration of the third sample audio is short and the pronunciation is unclear, the final timbre sub-model 102 obtained by training can still achieve a high degree of speech synthesis control capability for the target timbre.

[0112] Furthermore, the third sample audio has a target timbre. The third sample audio can be audio recorded by the user or audio with the desired timbre uploaded by the user. This disclosure does not limit the source or acquisition method of the third sample audio.

[0113] Specifically, the fourth acoustic feature corresponding to the third sample audio is input into the intermediate model to obtain the predicted Mel spectrum feature corresponding to the third sample audio output by the intermediate model; then, based on the second labeled Mel spectrum feature corresponding to the third sample audio and the predicted Mel spectrum feature corresponding to the third sample audio, the loss function information corresponding to this round of training is calculated; according to the loss function information, the coefficient values ​​of the parameters included in the intermediate model are adjusted to obtain the final timbre sub-model 102.

[0114] During the second phase of training, the second labeled Mel spectrum features corresponding to the third sample audio can be understood as the learning target of the intermediate model.

[0115] Based on the aforementioned introduction to prosodic sub-model 101, during the training process, if the fifth acoustic feature output by prosodic sub-model 101 based on the labeled text of the first input sample audio includes the predicted bottleneck feature, that is, prosodic sub-model 101 can achieve the mapping from text to bottleneck feature, then the third acoustic feature corresponding to the second sample audio input to timbre sub-model 102 includes the second labeled bottleneck feature corresponding to the second sample audio, and the fourth acoustic feature corresponding to the third sample audio input to the intermediate model includes the third labeled bottleneck feature corresponding to the third sample audio.

[0116] The second and third annotation bottleneck features can be obtained by extracting bottleneck features from the second and third sample audio samples respectively through the encoder of the ASR model. The implementation method is similar to that of obtaining the first annotation bottleneck features. For the sake of simplicity, it will not be described in detail here.

[0117] During training, if the prosody sub-model 101 outputs a fifth acoustic feature that includes a predicted bottleneck feature and a predicted fundamental frequency feature based on the labeled text of the first sample audio input, that is, the prosody sub-model 101 can achieve the mapping from text to bottleneck features and fundamental frequency features, then the third acoustic feature corresponding to the second sample audio input to the timbre sub-model 102 includes the second labeled bottleneck feature and the second labeled fundamental frequency feature corresponding to the second sample audio, and the fourth acoustic feature corresponding to the third sample audio input to the intermediate model includes the third labeled bottleneck feature and the third labeled fundamental frequency feature corresponding to the third sample audio.

[0118] The second and third annotation bottleneck features can be obtained by extracting bottleneck features from the second and third sample audio samples respectively using the encoder of the ASR model, similar to the method of obtaining the first annotation bottleneck features. The second and third annotation fundamental frequency features can be obtained by analyzing the second and third sample audio samples respectively using digital signal processing technology, similar to the method of obtaining the first annotation fundamental frequency features. For the sake of simplicity, they will not be elaborated here.

[0119] In summary, during the training process, the input of the timbre sub-model 102 and the output of the prosody sub-model 101 remain consistent.

[0120] Furthermore, when training the timbre sub-model 102, the initial values ​​of the coefficients corresponding to each parameter included in the timbre sub-model 102 can be preset or randomly initialized, and this disclosure does not limit this.

[0121] Furthermore, the loss functions corresponding to the timbre sub-models used in the first and second training phases can be the same or different, and this disclosure does not impose any restrictions on this.

[0122] in, Figure 1c An exemplary implementation of the timbre sub-model 102 is shown. (Refer to...) Figure 1c As shown, the timbre sub-model 102 can be implemented using a self-attention network structure.

[0123] Figure 1c In the timbre sub-model 102, there are: a convolutional network 1021 and one or more residual networks 1022. Each residual network 1022 includes: a self-attention network 1022a and a linear network 1022b.

[0124] The convolutional network 1021 is mainly used to perform convolution processing on the acoustic features corresponding to the input sample audio to model local feature information. The convolutional network 1021 may include one or more convolutional layers; this disclosure does not limit the number of convolutional layers included in the convolutional network 1021. Furthermore, the convolutional network 1021 inputs the local feature information to the connected residual network 1022.

[0125] The aforementioned one or more residual networks 1022, after passing through one or more residual networks 1022, are converted into spectral features (such as Mel spectral features).

[0126] It should be understood that intermediate models and Figure 1c The timbre sub-model 102 shown has the same structure, but the weight coefficients of the included parameters are not exactly the same.

[0127] By training the prosody sub-model 101 and timbre sub-model 102 respectively, a first feature extraction model and a second feature extraction model that meet the requirements of speech synthesis are finally obtained; then the first feature extraction model and the second feature extraction model are spliced ​​together to obtain a speech synthesis model that can synthesize the target timbre.

[0128] In some possible implementations, the speech synthesis model 100 may further include a vocoder 103. The vocoder 103 is used to convert the spectral features (such as Mel spectral features) output by the timbre sub-model 102 into audio. Of course, the vocoder can also be a standalone module, not bound to the speech synthesis model. Furthermore, this solution does not limit the specific type of vocoder.

[0129] In the above Figures 1a to 1c Based on the illustrated embodiment, the target speech synthesis model obtained through training has the ability to stably synthesize audio with the target timbre. Based on this, the target speech synthesis model can be used to process corresponding speech synthesis services.

[0130] Figure 2 A flowchart illustrating a speech synthesis method provided in an embodiment of this disclosure. (Refer to...) Figure 2 As shown, the speech synthesis method provided in this embodiment includes:

[0131] S201. Obtain the text to be processed.

[0132] The text to be processed may include one or more characters, or it may include one or more phonemes. The text to be processed is used to synthesize audio with the target rap style and target timbre.

[0133] This disclosure does not limit the method by which an electronic device acquires text to be processed.

[0134] For example, an electronic device can display a text input window and a soft keyboard to the user, who can then input text into the text input window by using the soft keyboard; alternatively, the user can input text into the text input window by copying and pasting; or the user can input an audio clip into the electronic device by speaking, and the electronic device can obtain the text to be processed by performing speech recognition on the user's audio input; or the user can import the file corresponding to the text to be processed into the electronic device, allowing the electronic device to obtain the text to be processed.

[0135] Users can input text to be processed into electronic devices in ways not limited to those described above. This is a simple and convenient operation that can enhance users' enthusiasm for creating multimedia content.

[0136] S202. Input the text to be processed into the speech synthesis model and obtain the spectral features corresponding to the text to be processed output by the speech synthesis model.

[0137] In some embodiments, the text to be processed is input into the speech synthesis model. The prosody sub-model extracts features from the text to be processed and outputs the first acoustic features corresponding to the text to be processed. The first acoustic features include bottleneck features corresponding to the text to be processed. The bottleneck features included in the first acoustic features are used to characterize the target rap style. The timbre sub-model receives the first acoustic features corresponding to the text to be processed as input and outputs the spectral features corresponding to the text to be processed.

[0138] In other embodiments, the text to be processed is input into the speech synthesis model. The prosody sub-model extracts features from the text to be processed and outputs the first acoustic features corresponding to the text to be processed. The first acoustic features include bottleneck features and fundamental frequency features corresponding to the text to be processed. The bottleneck features are used to characterize the target rap style, and the fundamental frequency features are used to characterize the pitch. The timbre sub-model receives the first acoustic features corresponding to the text to be processed as input and outputs the spectral features (such as Mel spectral features) corresponding to the text to be processed.

[0139] Among them, the speech synthesis model can be achieved through... Figures 1a to 1c The implementation method of the illustrated embodiment is as follows: the network structure of the speech synthesis model and the implementation method of training the speech synthesis model can be referred to the foregoing. Figures 1a to 1c The detailed description of the illustrated embodiments will not be repeated here for the sake of brevity.

[0140] In combination with the above Figure 1aAs shown in embodiment 1b, the text encoding network included in the prosodic sub-model can receive the text to be processed as input, analyze the context and temporal relationship of the text to be processed, and model the intermediate feature sequence; then, based on the attention coefficients output by the attention network included in the prosodic sub-model, the attention coefficients and the intermediate feature sequence are weighted and averaged to obtain a weighted average result; the decoding network included in the prosodic sub-model performs feature transformation on the input weighted average result and the output of the previous time step, and outputs the first acoustic feature corresponding to the text to be processed. The first acoustic feature may include the bottleneck feature corresponding to the text to be processed, or the first acoustic feature may include the bottleneck feature corresponding to the text to be processed and the fundamental frequency feature corresponding to the text to be processed.

[0141] In combination with the above Figure 1a As shown in embodiment 1c, the timbre sub-model includes a convolutional network that receives the first acoustic feature corresponding to the text to be processed as input, performs convolution processing on the first acoustic feature corresponding to the text to be processed, and models local feature information; the convolutional network inputs the local feature information to the connected residual network, and after passing through one or more residual networks, outputs the spectral feature (such as Mel spectral feature) corresponding to the text to be processed.

[0142] S203. Based on the spectral features of the text to be processed, obtain the target audio corresponding to the text to be processed. The target audio has the target timbre and the target rap style.

[0143] One possible implementation is that the electronic device can perform digital signal processing on the spectral features corresponding to the text to be processed based on a vocoder, thereby converting the spectral features corresponding to the text to be processed (such as the Mel spectral features corresponding to the text to be processed) into audio with the target timbre and the target rap style, i.e., target audio.

[0144] It should be noted that a vocoder can be used as part of a speech synthesis model, in which case the speech synthesis model can directly output audio with the target timbre and target rap style; in other cases, the vocoder can be used as an independent module outside the speech synthesis model, in which the vocoder can receive the spectral features corresponding to the text to be processed as input and convert the spectral features corresponding to the text to be processed into audio with the target timbre and target rap style.

[0145] The speech synthesis method provided in this embodiment analyzes the text to be processed based on a speech synthesis model and outputs the spectral features corresponding to the text. The speech synthesis model includes a prosody sub-model and a timbre sub-model. The prosody sub-model receives the text to be processed as input and outputs the first acoustic features corresponding to the text, which include bottleneck features representing the target rap style. The timbre sub-model receives the first acoustic features as input and outputs the spectral features corresponding to the text, which include information about the target timbre. By converting the spectral features output by the speech synthesis model, rap audio with the target rap style and timbre can be obtained, meeting the user's personalized audio needs. Furthermore, the speech synthesis model supports the conversion of any text to be processed, reducing the requirements for the user's music creation ability and encouraging the user to create multimedia content.

[0146] Figure 3 This is a schematic flowchart illustrating a speech synthesis method according to another embodiment of this disclosure. (Refer to...) Figure 3 As shown, the speech synthesis method provided in this embodiment is... Figure 2 Based on the illustrated embodiment, after step S203, which involves obtaining the target audio corresponding to the text to be processed based on the spectral features of the text to be processed, the following may also be included:

[0147] S204. Add the target audio corresponding to the text to be processed to the target multimedia content.

[0148] This disclosure does not limit the implementation method of adding target audio to target multimedia content. For example, when an electronic device adds target audio to target multimedia content, it can speed up or slow down the playback speed of the target audio by combining the duration of the target multimedia content and the duration of the target audio; it can also add subtitles corresponding to the target audio to the playback interface of the target multimedia content, or it can choose not to add subtitles corresponding to the target audio; if subtitles corresponding to the target audio are added to the playback interface of the target multimedia content, display parameters such as the color, font size, and font of the subtitles can also be set.

[0149] The method provided in this embodiment analyzes the text to be processed based on a speech synthesis model and outputs the spectral features corresponding to the text. The speech synthesis model includes a prosody sub-model and a timbre sub-model. The prosody sub-model receives the text to be processed as input and outputs a first acoustic feature corresponding to the text, which includes bottleneck features representing the target rap style. The timbre sub-model receives the first acoustic feature as input and outputs the spectral features corresponding to the text, which include spectral features representing the target timbre. By converting the spectral features output by the speech synthesis model, audio with the target rap style and target timbre can be obtained, meeting the user's personalized audio needs. Furthermore, the speech synthesis model supports the conversion of any text to be processed, reducing the requirements for the user's music creation ability and encouraging the user to create multimedia content.

[0150] In addition, adding the target audio to the target multimedia content makes the target multimedia content more interesting, thereby meeting the user's need to create creative videos.

[0151] By way of example, this disclosure also provides a speech synthesis apparatus.

[0152] Figure 4 This is a schematic diagram of a speech synthesis device provided according to an embodiment of the present disclosure. (Refer to...) Figure 4 As shown, the speech synthesis device 400 provided in this embodiment includes:

[0153] The acquisition module 401 is used to acquire the text to be processed.

[0154] The processing module 402 is used to input the text to be processed into a speech synthesis model and obtain the spectral features corresponding to the text to be processed output by the speech synthesis model; wherein, the speech synthesis model includes a prosody sub-model and a timbre sub-model, the prosody sub-model is used to output a first acoustic feature corresponding to the text to be processed based on the input text to be processed, the first acoustic feature including bottleneck features for characterizing the target rap style; the timbre sub-model is used to input the spectral features corresponding to the text to be processed based on the input first acoustic feature, the spectral features corresponding to the text to be processed including spectral features for characterizing the target timbre.

[0155] The processing module 402 is further configured to obtain the target audio corresponding to the text to be processed based on the spectral features corresponding to the text to be processed, wherein the target audio has the target timbre and the target rap style.

[0156] In some possible implementations, the prosodic sub-model is obtained by training based on the labeled text corresponding to the first sample audio and the second acoustic features corresponding to the first sample audio.

[0157] The first sample audio includes at least one audio of the target rap style; the second acoustic feature includes the first labeled bottleneck feature corresponding to the first sample audio.

[0158] In some possible implementations, the timbre sub-model is obtained by training based on the third acoustic feature corresponding to the second sample audio, the first labeled spectral feature corresponding to the second sample audio, the fourth acoustic feature corresponding to the third sample audio, and the second labeled spectral feature corresponding to the third sample audio.

[0159] The third acoustic feature includes the second labeled bottleneck feature corresponding to the second sample audio; the third sample audio includes at least one audio with the target timbre; and the fourth acoustic feature corresponding to the third sample audio includes the third labeled bottleneck feature corresponding to the third sample audio.

[0160] In some possible implementations, the first annotation bottleneck feature corresponding to the first sample audio, the second annotation bottleneck feature corresponding to the second sample audio, and the third annotation bottleneck feature corresponding to the third sample audio are obtained by the encoder of the end-to-end speech recognition model extracting bottleneck features from the input first sample audio, second sample audio, and third sample audio, respectively.

[0161] In some possible implementations, the second acoustic feature further includes: a first labeled fundamental frequency feature corresponding to the first sample audio;

[0162] The third acoustic feature further includes: the second labeled fundamental frequency feature corresponding to the second sample audio; the fourth acoustic feature further includes: the third labeled fundamental frequency feature corresponding to the third sample audio;

[0163] The first acoustic feature also includes: the fundamental frequency feature corresponding to the text to be processed.

[0164] In some possible implementations, the processing module 402 is also used to add the target audio corresponding to the text to be processed to the target multimedia content.

[0165] The speech synthesis device provided in this embodiment can be used to execute the technical methods of any of the above method embodiments. Its implementation principle and technical effect are similar, and can be referred to the detailed description of the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0166] By way of example, this disclosure also provides an electronic device.

[0167] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. (Refer to...) Figure 5As shown, the electronic device provided in this embodiment includes a memory 501 and a processor 502.

[0168] The memory 501 can be a separate physical unit, connected to the processor 502 via a bus 503. Alternatively, the memory 501 and processor 502 can be integrated together, implemented in hardware, etc.

[0169] The memory 501 is used to store program instructions, and the processor 502 calls the program instructions to execute the operations of any of the above method embodiments.

[0170] Optionally, when some or all of the methods in the above embodiments are implemented by software, the electronic device 500 may also include only the processor 502. The memory 501 for storing programs is located outside the electronic device 500, and the processor 502 is connected to the memory via circuits / wires to read and execute the programs stored in the memory.

[0171] The processor 502 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP.

[0172] The processor 502 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0173] The memory 501 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory may also include a combination of the above types of memory.

[0174] This disclosure also provides a readable storage medium, including: computer program instructions; when executed by at least one processor of an electronic device, the computer program instructions implement the speech synthesis method shown in any of the above method embodiments.

[0175] This disclosure also provides a program product comprising a computer program stored in a readable storage medium, wherein at least one processor of the electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the electronic device to implement the speech synthesis method as shown in any of the above method embodiments.

[0176] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech synthesis method, characterized in that, include: Get the text to be processed; The text to be processed is input into a speech synthesis model, and the spectral features corresponding to the text to be processed are obtained from the output of the speech synthesis model. The speech synthesis model includes a prosody sub-model and a timbre sub-model. The prosody sub-model outputs a first acoustic feature corresponding to the input text, which includes a bottleneck feature characterizing the target rap style. The timbre sub-model outputs a spectral feature corresponding to the input first acoustic feature, which includes a spectral feature characterizing the target timbre. The prosody sub-model and the timbre sub-model are decoupled from the speech synthesis model by introducing acoustic features including bottleneck features. Based on the spectral features corresponding to the text to be processed, the target audio corresponding to the text to be processed is obtained, and the target audio has the target timbre and the target rap style.

2. The method according to claim 1, characterized in that, The prosodic sub-model is obtained by training based on the labeled text corresponding to the first sample audio and the second acoustic features corresponding to the first sample audio. The first sample audio includes at least one audio of the target rap style; the second acoustic feature includes the first labeled bottleneck feature corresponding to the first sample audio.

3. The method according to claim 2, characterized in that, The timbre sub-model is obtained by training based on the third acoustic feature corresponding to the second sample audio, the first labeled spectral feature corresponding to the second sample audio, the fourth acoustic feature corresponding to the third sample audio, and the second labeled spectral feature corresponding to the third sample audio. The third acoustic feature includes the second labeled bottleneck feature corresponding to the second sample audio; the third sample audio includes at least one audio with the target timbre; and the fourth acoustic feature corresponding to the third sample audio includes the third labeled bottleneck feature corresponding to the third sample audio.

4. The method according to claim 3, characterized in that, The first annotation bottleneck feature corresponding to the first sample audio, the second annotation bottleneck feature corresponding to the second sample audio, and the third annotation bottleneck feature corresponding to the third sample audio are obtained by the encoder of the end-to-end speech recognition model extracting bottleneck features from the input first sample audio, second sample audio, and third sample audio respectively.

5. The method according to claim 3, characterized in that, The second acoustic feature further includes: the first labeled fundamental frequency feature corresponding to the first sample audio; The third acoustic feature further includes: the second labeled fundamental frequency feature corresponding to the second sample audio; the fourth acoustic feature further includes: the third labeled fundamental frequency feature corresponding to the third sample audio; The first acoustic feature also includes: the fundamental frequency feature corresponding to the text to be processed.

6. The method according to claim 1, characterized in that, The method further includes: Add the target audio corresponding to the text to be processed to the target multimedia content.

7. A speech synthesis device, characterized in that, include: The acquisition module is used to acquire the text to be processed; A processing module is used to input the text to be processed into a speech synthesis model and obtain the spectral features corresponding to the text to be processed output by the speech synthesis model. The speech synthesis model includes a prosody sub-model and a timbre sub-model. The prosody sub-model is used to output a first acoustic feature corresponding to the input text to be processed, the first acoustic feature including a bottleneck feature characterizing the target rap style, and the first acoustic feature including the bottleneck feature corresponding to the text to be processed. The timbre sub-model is used to output a spectral feature corresponding to the input first acoustic feature, the spectral feature corresponding to the text to be processed including a spectral feature characterizing the target timbre. The prosody sub-model and the timbre sub-model are obtained by decoupling the acoustic features including the bottleneck feature introduced by the speech synthesis model. The processing module is used to obtain the target audio corresponding to the text to be processed based on the spectral features of the text to be processed, wherein the target audio has the target timbre and the target rap style.

8. An electronic device, characterized in that, include: Memory, processor, and computer programs; The memory is configured to store the computer program; The processor is configured to execute the computer program to implement the speech synthesis method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, include: Computer program instructions; When the computer program instructions are executed by at least one processor of an electronic device, they implement the speech synthesis method as described in any one of claims 1 to 6.

10. A program product, characterized in that, include: Computer program instructions; The computer program is stored in a readable storage medium, and at least one processor of the electronic device reads the computer program instructions from the readable storage medium and executes the computer program instructions to implement the speech synthesis method as described in any one of claims 1 to 6.

Images