Speech synthesis method and device, electronic equipment and storage medium

Abstract

The application provides a speech synthesis method and device, electronic equipment and storage medium, and relates to the technical field of speech synthesis. The method introduces target attribute text in the speech synthesis process, can support speech synthesis with audio attributes corresponding to the target attribute text, and can control the expressiveness and rhythm of the target synthesized speech according to user needs, so that the target synthesized speech is more in line with user needs and user experience is improved. The speech synthesis model is trained by using the text sample with attribute labels, so that the speech synthesis model has the control ability of audio attributes during speech synthesis, can control the timbre, speech style, emotional expression, person setting, tone, rhythm and other audio attributes of the target synthesized speech, and can support language switching, environmental sound effects, dialect generation and other audio attributes, so as to ensure the generation quality of the speech synthesis model and improve the controllability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, and in particular to a speech synthesis method and device, an electronic device and a storage medium. BACKGROUND

[0002] Speech synthesis, also known as text-to-speech (TTS), aims to convert input text into fluent and natural output speech, and is a key technology for realizing intelligent human-computer voice interaction.

[0003] A mature TTS small model aims to synthesize clear speech with high sound quality and low errors, and pursues quality, with weaker prosodic performance and emotional performance of synthesized audio. A speech synthesis large model pursues to reduce the gap between synthesized speech and real communication speech, such as hoping to synthesize speech with large prosodic fluctuations and changes in an audiobook scenario, and hoping to synthesize speech with higher anthropomorphism and empathy in an interactive scenario, and so on, and the tolerance of pronunciation errors such as mouth pipe is improved.

[0004] Currently, mainstream speech synthesis large models mainly include an encoder and a decoder. In the training stage, phonemes and discrete acoustic features are spliced as inputs of the encoder, and the discrete acoustic features are taken as training targets, and an autoregressive and non-autoregressive combination is used for prediction; the predicted discrete acoustic features are converted into speech waveforms by the decoder. In the test stage, one scheme is to splice the phoneme sequence of a prompt, the phoneme sequence of a text to be synthesized, and the acoustic features of the prompt as inputs of the encoder, and to synthesize audio corresponding to the text to be synthesized with the habit and timbre of the prompt by the decoder. Another scheme is to input only the phoneme sequence of the text to be synthesized, and to synthesize audio with a random timbre by the decoder.

[0005] Although the synthesized audio obtained by using the above speech synthesis large model has greater improvement in richness and naturalness compared with the TTS small model, it only supports basic speech generation. If different languages, different styles, different dialects, etc. are needed to be synthesized, generally multiple independent models need to be trained, the process of synthesizing speech is complicated, and the cost of synthesizing speech is increased, and the efficiency of synthesizing speech is reduced. SUMMARY

[0006] The present application provides a speech synthesis method, device, electronic device and storage medium to solve the defects in the related art.

[0007] The present application provides a speech synthesis method, comprising:

[0008] obtain a target attribute text corresponding to the to-be-synthesized text;

[0009] input the to-be-synthesized text and the target attribute text into a speech synthesis model to obtain target synthesized speech corresponding to the to-be-synthesized text output by the speech synthesis model;

[0010] The speech synthesis model is trained based on multi-attribute speech samples, text samples with attribute labels corresponding to the multi-attribute speech samples, and discrete acoustic feature label sequences of the multi-attribute speech samples.

[0011] According to the speech synthesis method provided by the application, the speech synthesis model comprises a target encoder, a target encoding module and a target decoding module.

[0012] The inputting of the to-be-synthesized text and the target attribute text into the speech synthesis model to obtain the target synthesized speech corresponding to the to-be-synthesized text output by the speech synthesis model comprises:

[0013] fusing the to-be-synthesized text and the target attribute text to obtain a fused text;

[0014] inputting the fused text into the target encoder to obtain a first discrete acoustic feature corresponding to the to-be-synthesized text output by the target encoder;

[0015] inputting the fused text and the first discrete acoustic feature into the target encoding module to obtain a final discrete acoustic feature sequence determined by the target encoding module in an autoregressive manner;

[0016] inputting the to-be-synthesized text, the target attribute text and the final discrete acoustic feature sequence into the target decoding module to obtain the target synthesized speech output by the target decoding module.

[0017] According to the speech synthesis method provided by the application, the target decoding module comprises a first encoder, a second encoder and a decoder; and the inputting of the to-be-synthesized text, the target attribute text and the final discrete acoustic feature sequence into the target decoding module to obtain the target synthesized speech output by the target decoding module comprises:

[0018] inputting the to-be-synthesized text into the first encoder to obtain a first semantic feature output by the first encoder;

[0019] inputting the target attribute text into the second encoder to obtain a second semantic feature output by the second encoder;

[0020] inputting the first semantic feature, the second semantic feature, and the final discrete acoustic feature sequence to the decoder to obtain the target synthesized speech output by the decoder.

[0021] According to the speech synthesis method provided by the application, the target attribute text comprises a sentence-level attribute text and a local-level attribute text; the step of fusing the text to be synthesized with the target attribute text to obtain a fused text comprises:

[0022] The sentence-level attribute text is spliced with the text to be synthesized, and the local-level attribute text is inserted into the text to be synthesized to obtain the fused text.

[0023] According to the speech synthesis method provided by the application, the training step of the target encoding module comprises:

[0024] A pre-training encoding module is obtained, which is trained based on discrete acoustic feature data of multi-attribute speech data and multi-attribute text corresponding to attribute-free labels of the multi-attribute speech data;

[0025] The text sample and the first discrete acoustic feature sample corresponding to the text sample are input into the pre-training encoding module to obtain a final discrete acoustic feature sample sequence determined by the pre-training encoding module in an autoregressive manner;

[0026] Based on the final discrete acoustic feature sample sequence and the discrete acoustic feature label sequence, a first loss is calculated, and based on the first loss, the pre-training encoding module is iteratively trained to obtain the target encoding module.

[0027] According to the speech synthesis method provided by the application, the training step of the target decoding module comprises:

[0028] The non-attribute label and the attribute label in the text sample are respectively input into a first initial encoder and a second initial encoder of an initial decoding module to obtain a first initial semantic feature output by the first initial encoder and a second initial semantic feature output by the second initial encoder;

[0029] The first initial semantic feature, the second initial semantic feature, and the discrete acoustic feature label sequence are input into an initial decoder of the initial decoding module to obtain an initial synthesized speech output by the initial decoder;

[0030] Based on the initial synthesized speech and the multi-attribute speech sample, a second loss is calculated, and based on the second loss, the initial decoding module is iteratively trained to obtain the target decoding module.

[0031] According to the speech synthesis method provided by the application, the target attribute text includes at least one kind of audio attribute text of language, dialect, style, emotion, speaker gender, age, voice characteristics, body state, background audio and inserted audio.

[0032] The application further provides a speech synthesis device, comprising:

[0033] a text acquisition module, configured to acquire a text to be synthesized and a target attribute text;

[0034] a speech synthesis module, configured to input the text to be synthesized and the target attribute text into a speech synthesis model to obtain a target synthesized speech corresponding to the text to be synthesized output by the speech synthesis model;

[0035] The speech synthesis model is trained based on multi-attribute speech samples, text samples with attribute labels corresponding to the multi-attribute speech samples and discrete acoustic feature label sequences of the multi-attribute speech samples.

[0036] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech synthesis method according to any one of the above when executing the computer program.

[0037] The application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the speech synthesis method according to any one of the above.

[0038] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the speech synthesis method according to any one of the above.

[0039] The speech synthesis method, device, electronic device and storage medium provided by the application first acquire a text to be synthesized and a target attribute text, and then perform speech synthesis on the text to be synthesized by taking the target attribute text as a control signal through a speech synthesis model to obtain a target synthesized speech. The method introduces the target attribute text in the speech synthesis process, can support speech synthesis with audio attributes corresponding to the target attribute text, and can further control the expressiveness and rhythm of the target synthesized speech according to user requirements, so that the target synthesized speech is more in line with user requirements and user experience is improved. The speech synthesis model is trained by using text samples with attribute labels, so that the speech synthesis model has the control ability of audio attributes during speech synthesis, can control audio attributes such as timbre, speech style, emotional expression, person setting, tone, rhythm and the like of the target synthesized speech, and can support audio attributes such as language switching, environmental sound effects and dialect generation, thereby ensuring the generation quality of the speech synthesis model and improving the controllability. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0041] Figure 1 is a structural schematic diagram of an existing speech synthesis large model.

[0042] Figure 2 is a flowchart of a speech synthesis method provided by the present application.

[0043] Figure 3 is a structural schematic diagram of a target decoding module provided by the present application.

[0044] Figure 4 is an input and output schematic diagram of a target encoding module provided by the present application.

[0045] Figure 5 is an input and output schematic diagram of two stages in the training process of a target encoding module provided by the present application.

[0046] Figure 6 is a structural schematic diagram of a speech synthesis device provided by the present application.

[0047] Figure 7 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0049] As shown in Figure 1 , the existing speech synthesis large model usually includes an encoder and a decoder. The input of the encoder includes the phoneme sequence of the text to be synthesized and the current acoustic feature sequence. The output of the encoder is the acoustic feature of the next frame, which is used to update the current acoustic feature sequence. The updated acoustic feature sequence can be marked by a start identifier (start_id) and an end identifier (end_id). Finally, the final acoustic feature sequence output by the encoder is input to the decoder, which is converted into an audio waveform by the decoder.

[0050] Compared with a small TTS model, an existing large voice synthesis model has great improvement in richness and naturalness of synthesized audio, but it only supports basic voice generation, and multiple independent models need to be trained to simultaneously realize voice synthesis of different languages, different styles, different dialects and the like, resulting in a complicated process of synthesized voice and increased cost and reduced efficiency of synthesized voice. Based on this, a voice synthesis method is provided in the embodiments of the present application.

[0051] Figure 2 A flowchart of the voice synthesis method provided in the embodiments of the present application is shown in Figure 2 The method comprises:

[0052] S1, obtaining a text to be synthesized and a target attribute text;

[0053] S2, inputting the text to be synthesized and the target attribute text into a voice synthesis model to obtain a target synthesized voice corresponding to the text to be synthesized output by the voice synthesis model;

[0054] The voice synthesis model is trained based on a multi-attribute voice sample, a text sample with attribute labels corresponding to the multi-attribute voice sample and a discrete acoustic feature label sequence of the multi-attribute voice sample.

[0055] Specifically, the voice synthesis method provided in the embodiments of the present application has a voice synthesis device as an execution subject, which can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet and the like, which is not specifically limited here.

[0056] First, step S1 is performed to obtain a text to be synthesized and a target attribute text provided by a user. The text to be synthesized refers to a text that needs to be used to generate a target synthesized voice, and the text to be synthesized is used as the text content of the generated target synthesized voice.

[0057] The target attribute text is used to describe at least one type of audio attribute of the target synthesized voice, such as language, dialect, voice style, emotional expression, speaker gender, age, voice characteristics, body state, background sound and inserted sound.

[0058] Then, step S2 is performed to input the text to be synthesized and the target attribute text into a voice synthesis model, which can be a large language model (LLM). The voice synthesis model performs voice synthesis on the text to be synthesized by taking the target attribute text as a control text, and obtains and outputs the target synthesized voice corresponding to the text to be synthesized.

[0059] Here, the speech synthesis model can be obtained by training an initial synthesis model using multi-attribute speech samples, text samples with attribute labels corresponding to the multi-attribute speech samples, and discrete acoustic feature label sequences of the multi-attribute speech samples. The multi-attribute speech samples can be speech data of different languages, different dialects, different speech styles, different emotional expressions obtained by recording, or audio and video data of different sources collected.

[0060] The text sample is a text obtained by transcribing the multi-attribute speech sample and annotated with an attribute label, and the text sample is the text content of the multi-attribute speech sample. The multi-attribute speech sample corresponds to a known discrete acoustic feature label sequence. The discrete acoustic feature label sequence is a sequence obtained by arranging the discrete acoustic features of each frame of speech in the multi-attribute speech sample in chronological order. Here, the discrete acoustic features can be discrete audio codes.

[0061] A feasible training scheme for the initial synthesis model can include:

[0062] The text sample and the discrete acoustic feature label sequence are input into the initial synthesis model to obtain a synthesis result output by the initial synthesis model. Then, a loss function value is calculated according to the synthesis result and the multi-attribute speech sample. Finally, the model parameters of the initial synthesis model are updated according to the loss function value. The input process and the calculation process are iteratively performed until the loss function converges or a preset number of iterations is reached, and a speech synthesis model is obtained. The preset number of iterations can be set as needed, and is not limited herein.

[0063] The speech synthesis method provided in the embodiments of the present application first acquires a text to be synthesized and a target attribute text. Then, the speech synthesis model is used to synthesize speech for the text to be synthesized by taking the target attribute text as a control signal, and a target synthesized speech is obtained. This method introduces a target attribute text in the speech synthesis process, which can support speech synthesis with audio attributes corresponding to the target attribute text, and can control the expressiveness and rhythm of the target synthesized speech according to user needs, so that the target synthesized speech is more in line with user needs and improves user experience. The speech synthesis model is trained using text samples with attribute labels, which can enable the speech synthesis model to have control ability of audio attributes during speech synthesis, and can control the timbre, speech style, emotional expression, persona, tone, rhythm, and other audio attributes of the target synthesized speech, and can support language switching, environmental sound effects, dialect generation, and other audio attributes, to ensure the quality of the speech synthesis model while improving controllability.

[0064] Further, it is ensured that the speech synthesis has audio attributes that meet user needs.

[0065] On the basis of the above-mentioned embodiments, the speech synthesis model comprises a target encoder, a target encoding module and a target decoding module;

[0066] The inputting of the to-be-synthesized text and the target attribute text into the speech synthesis model comprises:

[0067] The to-be-synthesized text and the target attribute text are fused to obtain a fused text;

[0068] The inputting of the to-be-synthesized text and the target attribute text into the speech synthesis model comprises:

[0069] The inputting of the to-be-synthesized text and the target attribute text into the speech synthesis model comprises:

[0070] The inputting of the to-be-synthesized text and the target attribute text into the speech synthesis model comprises:

[0071] Specifically, before the inputting of the to-be-synthesized text and the target attribute text into the speech synthesis model, the to-be-synthesized text and the target attribute text can be fused to obtain a fused text. Here, the fusion manner can be splicing, and the splicing position can be determined according to actual conditions.

[0072] In the embodiments of the present application, the speech synthesis model comprises a target encoder, a target encoding module and a target decoding module, and the inputting of the fused text into the target encoder can generate the first discrete acoustic feature corresponding to the to-be-synthesized text, i.e. the first discrete acoustic feature corresponding to the target synthesized speech corresponding to the to-be-synthesized text.

[0073] After that, the fusion text and the first discrete acoustic feature are input into a target encoding module, and the target encoding module can iteratively determine a final discrete acoustic feature sequence corresponding to the text to be synthesized in an autoregressive manner. For example, the target encoding module can predict a second discrete acoustic feature according to the fusion text and the first discrete acoustic feature, and then the target encoding module can continue to predict a third discrete acoustic feature according to the fusion text and the existing discrete acoustic feature sequence, i.e., the first discrete acoustic feature and the second discrete acoustic feature, and the target encoding module can continue to predict a fourth discrete acoustic feature according to the fusion text and the existing discrete acoustic feature sequence, i.e., the first discrete acoustic feature, the second discrete acoustic feature and the third discrete acoustic feature, and so on, until the last discrete acoustic feature is predicted. Finally, all the obtained discrete acoustic features can be arranged in time sequence to form the final discrete acoustic feature sequence.

[0074] It can be understood that the input of the target encoding module includes the fusion text and the discrete acoustic feature sequence, and in particular, the first input discrete acoustic feature sequence only includes the first discrete acoustic feature.

[0075] After that, the text to be synthesized, the target attribute text and the final discrete acoustic feature sequence are input into a target decoding module, and the target decoding module can convert the final discrete acoustic feature sequence into an audio waveform to obtain the target synthesized speech in combination with the text to be synthesized and the target attribute text.

[0076] In the embodiment of the present application, the target encoding module determines the final discrete acoustic feature sequence in an autoregressive manner, and each time only the fusion text and the historical discrete acoustic features determined need to be input to predict the subsequent discrete acoustic features, without the need for additional variables or complex data collection processes. Moreover, the processing process of the target decoding module introduces the text to be synthesized and the target attribute text, which can provide a reference for the processing process and reduce the difficulty of the processing process. The specific structure and implementation logic of the speech synthesis model can reduce the difficulty of speech synthesis and improve the efficiency of speech synthesis.

[0077] On the basis of the above-mentioned embodiment, the target decoding module includes a first encoder, a second encoder and a decoder; and the input of the text to be synthesized, the target attribute text and the final discrete acoustic feature sequence into the target decoding module to obtain the target synthesized speech output by the target decoding module includes:

[0078] The text to be synthesized is input into the first encoder to obtain the first semantic feature output by the first encoder;

[0079] The target attribute text is input into the second encoder to obtain the second semantic feature output by the second encoder;

[0080] input the first semantic feature, the second semantic feature and the final discrete acoustic feature sequence to the decoder to obtain the target synthesized speech output by the decoder.

[0081] Specifically, in the embodiment of the present application, as shown in the figure, Figure 3 The target decoding module can include a first encoder, a second encoder and a decoder.

[0082] In the process of determining the target synthesized speech, the target decoding module can first input the text to be synthesized and the target attribute text to the first encoder and the second encoder respectively, extract the first semantic feature of the text to be synthesized through the first encoder, and extract the second semantic feature of the target attribute text through the second encoder.

[0083] Thereafter, the first semantic feature, the second semantic feature and the final discrete acoustic feature sequence are input to the decoder, and the decoder converts the final discrete acoustic feature sequence into the target synthesized speech using the first semantic feature and the second semantic feature.

[0084] In the embodiment of the present application, by introducing the first encoder and the second encoder into the target decoding module, the text to be synthesized and the target attribute text are converted into semantic features, which facilitates the conversion process of the final discrete acoustic feature sequence, and the obtained target synthesized speech is more consistent with the semantics of the text to be synthesized and the target attribute text.

[0085] On the basis of the above-mentioned embodiment, the target attribute text includes sentence-level attribute text and local-level attribute text; and the fusion of the text to be synthesized and the target attribute text to obtain a fusion text includes:

[0086] The sentence-level attribute text is spliced with the text to be synthesized, and the local-level attribute text is inserted into the text to be synthesized to obtain the fusion text.

[0087] Specifically, the target attribute text used in the embodiment of the present application can include sentence-level attribute text and local-level attribute text. The sentence-level attribute text is used to describe the audio attributes applicable to the entire target synthesized speech, and the local-level attribute text is used to describe the audio attributes corresponding to the text near the position of the local-level attribute text in the text to be synthesized.

[0088] In the fusion of the to-be-synthesized text and the target attribute text, the sentence-level attribute text can be directly spliced before the to-be-synthesized text, and the local-level attribute text can be inserted into the to-be-synthesized text at a corresponding position to obtain a fusion text. The to-be-synthesized text and each target attribute text can be separated by an identifier. The identifier can include a start identifier and an end identifier. Meanwhile, when the fusion text and each discrete acoustic feature sequence are input into the target encoding module, the fusion text and the discrete acoustic feature sequence can also be spliced and separated by an identifier.

[0089] For example, as shown in FIG. 6, the input of the target encoding module can be represented as: Figure 4 <attr><Chinese>One wise old man, with low and steady voice, not in a hurry to tell the past< / attr> <Bos_t>Remember when, the old husband is in prime, out of the door travel encounter <coughing sound> injustice, must be on the road and see injustice draw a sword to help <Eos_t> <Bos_a>discrete acoustic feature sequence <Eos_a>, wherein “ <attr>"and"< / attr> ” are a start identifier and an end identifier of the sentence-level attribute text respectively, “<coughing sound>” is an identifier of the local-level attribute text, indicating that a coughing sound appears at a corresponding position in the target synthesized audio, “<Chinese>” is the language of the to-be-synthesized text, “<Bos_t>” and “<Eos_t>” are a start identifier and an end identifier of the to-be-synthesized text respectively, and “<Bos_a>” and “<Eos_a>” are a start identifier and an end identifier of the discrete acoustic feature sequence respectively.

[0090] Figure 4 In the above embodiment, the discrete acoustic feature sequence can also contain a speaker identification (spkid) for distinguishing discrete acoustic feature sequences of different speakers. It should be noted that, Figure 4 In the above embodiment, the discrete acoustic feature sequence contains the discrete acoustic feature sequence currently input into the target encoding module and the next discrete acoustic feature output by the target encoding module.

[0091] On the basis of the above embodiment, the training step of the target encoding module comprises:

[0092] obtaining a pre-trained encoding module, wherein the pre-trained encoding module is trained based on discrete acoustic feature data of multi-attribute speech data and multi-attribute text corresponding to attribute-free labels of the multi-attribute speech data;

[0093] inputting the text sample and the first discrete acoustic feature sample corresponding to the text sample into the pre-trained encoding module to obtain a final discrete acoustic feature sample sequence determined by the pre-trained encoding module in an autoregressive manner;

[0094] ​Based on the final discrete acoustic feature sample sequence and the discrete acoustic feature label sequence, a first loss is calculated, and based on the first loss, the pre-training encoding module is iteratively trained to obtain the target encoding module.

[0095] Specifically, in the embodiments of the present application, the target encoding module and the target decoding module in the speech synthesis model can be independently trained or jointly trained. When the target encoding module is trained independently, a pre-training encoding module can be obtained first, which can be obtained by training an initial encoding module based on the discrete acoustic feature data of the multi-attribute speech data and the multi-attribute text corresponding to the attribute-free label of the multi-attribute speech data. The multi-attribute speech data can be speech data covering enough audio attributes such as languages, dialects, styles, etc., and can include millions of hours of speech data to ensure that the pre-training encoding module has strong general speech synthesis capability.

[0096] The discrete acoustic feature data of the multi-attribute speech data can be obtained by inputting the multi-attribute speech data into the encoder of Encodec.

[0097] During the training process of the pre-training encoding module, the multi-attribute text can be input into the initial encoding module first to obtain the output result of the initial encoding module. Based on the output result of the initial encoding module and the discrete acoustic feature data, a loss function value is calculated, and based on the loss function value, the model parameters of the initial encoding model are updated. The input process and the calculation process are iteratively performed until the loss function converges or a specified number of iterations is reached, and the pre-training encoding module is obtained. The specified number of iterations can be set as needed, which is not limited here.

[0098] Thereafter, the text sample and the first discrete acoustic feature sample corresponding to the text sample are input into the pre-training encoding module to obtain a final discrete acoustic feature sample sequence determined by the pre-training encoding module in an autoregressive manner. For example, the pre-training encoding module can predict the second discrete acoustic feature sample based on the text sample and the first discrete acoustic feature sample. Thereafter, the pre-training encoding module can continue to predict the third discrete acoustic feature sample based on the text sample and the existing discrete acoustic feature sample sequence, i.e., the first discrete acoustic feature sample and the second discrete acoustic feature sample. The pre-training encoding module can continue to predict the fourth discrete acoustic feature sample based on the text sample and the existing discrete acoustic feature sample sequence, i.e., the first discrete acoustic feature sample, the second discrete acoustic feature sample and the third discrete acoustic feature sample. This process continues until the last discrete acoustic feature sample is predicted. Finally, all the discrete acoustic feature samples obtained can be arranged in chronological order to form the final discrete acoustic feature sample sequence.

[0099] After that, by using the final discrete acoustic feature sample sequence and the discrete acoustic feature label sequence, the first loss can be calculated by the following formula:

[0100] ;

[0101] wherein P is the first loss, represents the discrete acoustic feature sample sequence containing t+1 discrete acoustic feature samples, represents the discrete acoustic feature sample sequence containing t discrete acoustic feature samples, T represents the difference between the number of discrete acoustic feature samples in the final discrete acoustic feature sample sequence and 1, and x represents the text sample containing attribute labels. is the model parameter of the pre-training encoding module, represents the use of and the corresponding subsequence in the discrete acoustic feature label sequence to calculate the bias.

[0102] Finally, by using the first loss, the model parameter of the pre-training encoding module is iteratively trained, and when the first target iteration number is reached or the first loss converges, the target encoding module is obtained.

[0103] As shown in Figure 5 , the training process of the target encoding module can include two stages, the first stage is the process of training the initial encoding module to obtain the pre-training encoding module, and the second stage is the process of training the pre-training encoding module to obtain the target encoding module.

[0104] In the embodiment of the application, the pre-training encoding module is first trained by using the discrete acoustic feature data and the multi-attribute text, and then the pre-training encoding module is trained, which can reduce the requirement for the number of text samples and further reduce the model training cost.

[0105] On the basis of the above embodiment, the training step of the target decoding module comprises:

[0106] The non-attribute labels and the attribute labels in the text sample are respectively input into the first initial encoder and the second initial encoder of the initial decoding module to obtain the first initial semantic features output by the first initial encoder and the second initial semantic features output by the second initial encoder.

[0107] The first initial semantic features, the second initial semantic features, and the discrete acoustic feature label sequence are input into the initial decoder of the initial decoding module to obtain the initial synthesized speech output by the initial decoder.

[0108] Based on the initial synthesized speech and the multi-attribute speech sample, a second loss is calculated, and the initial decoding module is iteratively trained based on the second loss to obtain the target decoding module.

[0109] Specifically, the text sample contains non-attribute labels and attribute labels, and the non-attribute label refers to the text in the text sample other than the attribute label. When training the target decoding module alone, the non-attribute label and the attribute label in the text sample can be input into the first initial encoder and the second initial encoder of the initial decoding module respectively to obtain the first initial semantic feature output by the first initial encoder and the second initial semantic feature output by the second initial encoder.

[0110] Then, the first initial semantic feature, the second initial semantic feature and the discrete acoustic feature label sequence are input into the initial decoder of the initial decoding module to obtain the initial synthesized speech output by the initial decoder.

[0111] Finally, the initial synthesized speech and the multi-attribute speech sample are used to calculate a second loss, and the model parameters of the initial decoding module are iteratively trained using the second loss. When the second target iteration number is reached or the second loss converges, the target decoding module is obtained.

[0112] In the embodiment of the application, the first initial encoder, the second initial encoder and the initial decoder in the initial decoding module are jointly trained, which can improve the training efficiency of the target decoding module and save training time.

[0113] As shown in the above embodiment, the embodiment of the application provides a speech synthesis device, which comprises: Figure 6

[0114] The text acquisition module 61 is configured to acquire a text to be synthesized and a target attribute text.

[0115] The speech synthesis module 62 is configured to input the text to be synthesized and the target attribute text into a speech synthesis model to obtain a target synthesized speech corresponding to the text to be synthesized output by the speech synthesis model.

[0116] The speech synthesis model is trained based on a multi-attribute speech sample, a text sample with attribute labels corresponding to the multi-attribute speech sample and a discrete acoustic feature label sequence of the multi-attribute speech sample.

[0117] On the basis of the above embodiment, the speech synthesis device provided in the embodiment of the application comprises a target encoder, a target encoding module and a target decoding module.

[0118] The speech synthesis module is specifically configured to: ​

[0119] fuse the text to be synthesized with the target attribute text to obtain a fusion text;

[0120] input the fusion text to the target encoder to obtain a first discrete acoustic feature corresponding to the text to be synthesized output by the target encoder;

[0121] input the fusion text and the first discrete acoustic feature to the target encoding module to obtain a final discrete acoustic feature sequence determined by the target encoding module in an autoregressive manner;

[0122] input the text to be synthesized, the target attribute text, and the final discrete acoustic feature sequence to the target decoding module to obtain the target synthesized speech output by the target decoding module.

[0123] In the above embodiments, the speech synthesis device provided in the embodiments of the present application comprises a first encoder, a second encoder, and a decoder.

[0124] The speech synthesis module is further specifically configured to:

[0125] input the text to be synthesized to the first encoder to obtain a first semantic feature output by the first encoder;

[0126] input the target attribute text to the second encoder to obtain a second semantic feature output by the second encoder;

[0127] input the first semantic feature, the second semantic feature, and the final discrete acoustic feature sequence to the decoder to obtain the target synthesized speech output by the decoder.

[0128] In the above embodiments, the speech synthesis device provided in the embodiments of the present application, the target attribute text comprises a sentence-level attribute text and a local-level attribute text.

[0129] The speech synthesis module is further specifically configured to:

[0130] splice the sentence-level attribute text with the text to be synthesized, and insert the local-level attribute text into the text to be synthesized to obtain the fusion text.

[0131] In the above embodiments, the speech synthesis device provided in the embodiments of the present application further comprises a first training module configured to:

[0132] obtain a pre-trained encoding module, wherein the pre-trained encoding module is trained based on discrete acoustic feature data of multi-attribute speech data and multi-attribute text corresponding to attribute-free labels of the multi-attribute speech data.

[0133] input the text sample and the first discrete acoustic feature sample corresponding to the text sample into a pre-training encoding module to obtain a final discrete acoustic feature sample sequence determined by the pre-training encoding module in an autoregressive manner;

[0134] based on the final discrete acoustic feature sample sequence and the discrete acoustic feature label sequence, calculate a first loss, and based on the first loss, iteratively train the pre-training encoding module to obtain the target encoding module.

[0135] On the basis of the above-mentioned embodiments, the speech synthesis device provided in the embodiments of the present application further comprises a second training module configured to:

[0136] input the non-attribute label in the text sample and the attribute label into a first initial encoder and a second initial encoder of an initial decoding module respectively to obtain a first initial semantic feature output by the first initial encoder and a second initial semantic feature output by the second initial encoder;

[0137] input the first initial semantic feature, the second initial semantic feature and the discrete acoustic feature label sequence into an initial decoder of the initial decoding module to obtain an initial synthesized speech output by the initial decoder;

[0138] based on the initial synthesized speech and the multi-attribute speech sample, calculate a second loss, and based on the second loss, iteratively train the initial decoding module to obtain the target decoding module.

[0139] On the basis of the above-mentioned embodiments, the speech synthesis device provided in the embodiments of the present application, the target attribute text includes at least one kind of audio attribute text of language, dialect, style, emotion, pronunciation person gender, age, voice characteristics, physical state, background audio and inserted audio.

[0140] Specifically, the roles of each module in the speech synthesis device provided in the embodiments of the present application are one-to-one corresponding to the operation processes of each step in the method embodiment, and the effects achieved are consistent. For details, refer to the above-mentioned embodiments, which will not be repeated here.

[0141] Figure 7 An example of an entity structure schematic diagram of an electronic device is shown as Figure 7As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete communications with each other through the communications bus 840. The processor 810 can invoke a logical instruction in the memory 830 to execute the speech synthesis method provided in each of the embodiments described above.

[0142] In addition, the logical instruction in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the related art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0143] On the other hand, the present application also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium, and the computer program is executed by a processor, so that the computer can execute the speech synthesis method provided in each of the embodiments described above.

[0144] In yet another aspect, the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the speech synthesis method provided in each of the embodiments described above. The computer-readable storage medium can be a non-transitory computer-readable storage medium or a transitory computer-readable storage medium, which is not specifically limited here.

[0145] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0146] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and the necessary general hardware platform from the above description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0147] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the text to be synthesized and the target attribute text; The text to be synthesized and the target attribute text are input into the speech synthesis model to obtain the target synthesized speech corresponding to the text to be synthesized output by the speech synthesis model. The speech synthesis model is trained based on multi-attribute speech samples, text samples with attribute labels corresponding to the multi-attribute speech samples, and discrete acoustic feature label sequences of the multi-attribute speech samples. The speech synthesis model includes a target encoder, a target encoding module, and a target decoding module; The step of inputting the text to be synthesized and the target attribute text into the speech synthesis model to obtain the target synthesized speech corresponding to the text to be synthesized output by the speech synthesis model includes: The text to be synthesized is merged with the target attribute text to obtain the merged text; The fused text is input into the target encoder to obtain the first discrete acoustic feature corresponding to the text to be synthesized, which is output by the target encoder. The fused text and the first discrete acoustic feature are input into the target encoding module to obtain the final discrete acoustic feature sequence determined by the target encoding module using an autoregressive approach; The text to be synthesized, the target attribute text, and the final discrete acoustic feature sequence are input into the target decoding module to obtain the target synthesized speech output by the target decoding module.

2. The speech synthesis method according to claim 1, characterized in that, The target decoding module includes a first encoder, a second encoder, and a decoder; the step of inputting the text to be synthesized, the target attribute text, and the final discrete acoustic feature sequence into the target decoding module to obtain the target synthesized speech output by the target decoding module includes: The text to be synthesized is input into the first encoder to obtain the first semantic feature output by the first encoder; The target attribute text is input into the second encoder to obtain the second semantic feature output by the second encoder; The first semantic feature, the second semantic feature, and the final discrete acoustic feature sequence are input into the decoder to obtain the target synthesized speech output by the decoder.

3. The speech synthesis method according to claim 1, characterized in that, The target attribute text includes sentence-level attribute text and local-level attribute text; the process of fusing the text to be synthesized with the target attribute text to obtain fused text includes: The sentence-level attribute text is concatenated with the text to be synthesized, and the local-level attribute text is inserted into the text to be synthesized to obtain the fused text.

4. The speech synthesis method according to claim 1, characterized in that, The training steps for the target encoding module include: A pre-trained encoding module is obtained, which is trained based on the discrete acoustic feature data of multi-attribute speech data and the multi-attribute text without attribute labels corresponding to the multi-attribute speech data. The text sample and the first discrete acoustic feature sample corresponding to the text sample are input into the pre-trained encoding module to obtain the final discrete acoustic feature sample sequence determined by the pre-trained encoding module using an autoregressive method; Based on the final discrete acoustic feature sample sequence and the discrete acoustic feature label sequence, a first loss is calculated, and based on the first loss, the pre-trained coding module is iteratively trained to obtain the target coding module.

5. The speech synthesis method according to claim 2, characterized in that, The training steps for the target decoding module include: The non-attribute tags and attribute tags in the text sample are respectively input to the first initial encoder and the second initial encoder of the initial decoding module to obtain the first initial semantic feature output by the first initial encoder and the second initial semantic feature output by the second initial encoder. The first initial semantic feature, the second initial semantic feature, and the discrete acoustic feature label sequence are input into the initial decoder of the initial decoding module to obtain the initial synthesized speech output by the initial decoder; Based on the initial synthesized speech and the multi-attribute speech samples, a second loss is calculated, and based on the second loss, the initial decoding module is iteratively trained to obtain the target decoding module.

6. The speech synthesis method according to any one of claims 1-5, characterized in that, The target attribute text includes text with at least one type of audio attribute, such as language, dialect, style, emotion, speaker gender, age, voice characteristics, physical condition, background audio, and inserted audio.

7. A speech synthesis device, characterized in that, include: The text acquisition module is used to acquire the text to be synthesized and the target attribute text; The speech synthesis module is used to input the text to be synthesized and the target attribute text into the speech synthesis model to obtain the target synthesized speech corresponding to the text to be synthesized output by the speech synthesis model. The speech synthesis model is trained based on multi-attribute speech samples, text samples with attribute labels corresponding to the multi-attribute speech samples, and discrete acoustic feature label sequences of the multi-attribute speech samples. The speech synthesis model includes a target encoder, a target encoding module, and a target decoding module; The step of inputting the text to be synthesized and the target attribute text into the speech synthesis model to obtain the target synthesized speech corresponding to the text to be synthesized output by the speech synthesis model includes: The text to be synthesized is merged with the target attribute text to obtain the merged text; The fused text is input into the target encoder to obtain the first discrete acoustic feature corresponding to the text to be synthesized, which is output by the target encoder. The fused text and the first discrete acoustic feature are input into the target encoding module to obtain the final discrete acoustic feature sequence determined by the target encoding module using an autoregressive approach; The text to be synthesized, the target attribute text, and the final discrete acoustic feature sequence are input into the target decoding module to obtain the target synthesized speech output by the target decoding module.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech synthesis method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method as described in any one of claims 1-6.

Images