A speech synthesis method, device, electronic device and storage medium

Abstract

The present invention provides a speech synthesis method, device, electronic device and storage medium, which relate to the field of artificial intelligence. The method comprises: obtaining a text to be synthesized and a timbre reference audio; inputting the text to be synthesized into a pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is trained using training text and corresponding training semantic features, and the training semantic features are generated using training speech audio corresponding to the training text, and the training speech audio has a preset speech style; the semantic features, the text to be synthesized and the timbre reference audio are input into the pre-trained speech synthesis model for speech synthesis, and synthesized speech audio is obtained; the semantic features including the speech content and speech style can be used together with the text to be synthesized and the timbre reference audio for speech synthesis, so as to achieve timbre style decoupling.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a speech synthesis method, device, electronic equipment and storage medium. Background Art

[0002] Speech synthesis technology refers to the technology of converting text content into speech audio. With the continuous development of artificial intelligence technology, especially with the continuous advancement of large-scale speech synthesis models, the effect of speech synthesis is improving day by day. However, in related technologies, although the speech synthesis model can directly convert text content into speech audio with a specific timbre, it cannot independently adjust the speech style of the speech audio generated by the model, thereby reducing the flexibility of speech synthesis. Summary of the invention

[0003] The purpose of the present invention is to provide a speech synthesis method, device, electronic device and storage medium, which can use semantic features (semantic tokens) containing speech content and speech style together with the text to be synthesized and the timbre reference audio to perform speech synthesis, so as to achieve decoupling of speech timbre and style.

[0004] In order to solve the above technical problems, the present invention provides a speech synthesis method, comprising:

[0005] Obtain the text to be synthesized and the timbre reference audio;

[0006] Inputting the text to be synthesized into a pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is trained using the training text and the corresponding training semantic features, the training semantic features are generated using the training speech audio corresponding to the training text, and the training speech audio has a preset speech style;

[0007] The semantic features, the text to be synthesized and the timbre reference audio are input into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio.

[0008] Optionally, the pre-trained decoding model is an autoregressive decoding model;

[0009] The step of inputting the text to be synthesized into a pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized, including:

[0010] Inputting the text to be synthesized into the pre-trained decoding model to obtain a first semantic feature;

[0011] Determine whether the number of semantic features currently generated reaches a preset value; wherein the preset value is set according to a preset duration of the synthesized speech audio;

[0012] If not, the text to be synthesized and all the semantic features currently generated are input into the pre-trained decoding model to obtain the next semantic feature, and then the step of determining whether the number of semantic features currently generated reaches a preset value is entered;

[0013] If yes, stop generating semantic features.

[0014] Optionally, the training process of the pre-trained decoding model includes:

[0015] Acquire the training text and the training voice audio;

[0016] Extracting features from the training speech audio to obtain audio features, and quantizing the audio features to obtain a training semantic feature sequence;

[0017] The initial decoding model is trained using the training text and the training semantic feature sequence to obtain the pre-trained decoding model.

[0018] Optionally, the training process of the initial decoding model using the training text and the training semantic feature sequence to obtain the pre-trained decoding model includes:

[0019] Inputting the training text into the initial decoding model to obtain a semantic feature sequence to be tested;

[0020] Determining the loss value between the training semantic feature sequence and the semantic feature sequence to be tested by using a preset loss function;

[0021] The loss value is used to update the parameters of the initial decoding model to obtain the pre-trained decoding model.

[0022] Optionally, the pre-trained speech synthesis model includes an attention module, a timbre encoder and a decoder;

[0023] The step of inputting the semantic features, the text to be synthesized and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio comprises:

[0024] Inputting the semantic feature and the text to be synthesized into the attention module, so that the attention module performs attention processing on the semantic feature using the text to be synthesized to obtain an enhanced semantic feature;

[0025] Converting the timbre reference audio into a linear spectrum, and inputting the linear spectrum into the timbre encoder to obtain timbre representation information;

[0026] The enhanced semantic features and the timbre representation information are input into the decoder to obtain the synthesized speech audio.

[0027] Optionally, the training process of the pre-trained speech synthesis model includes:

[0028] Acquire the training text, the training speech audio, the training semantic features and the training timbre reference audio;

[0029] Inputting the training semantic features and the training text into the attention module, so that the attention module performs attention processing on the training semantic features using the training text to obtain training enhanced semantic features;

[0030] Converting the training timbre reference audio into a linear spectrum, and inputting the linear spectrum into the timbre encoder to obtain training timbre representation information;

[0031] Inputting the training enhanced semantic features and the training timbre representation information into the decoder to obtain the synthesized speech audio to be tested;

[0032] The loss value is calculated using the training speech audio and the synthesized speech audio to be tested, and the attention module, the timbre encoder and the decoder are updated using the loss value to obtain the pre-trained speech synthesis model.

[0033] Optionally, the acquiring of training semantic features includes:

[0034] Using a pre-trained audio feature extraction model to extract features from the training speech audio to obtain audio features;

[0035] The audio features are quantized to obtain the training semantic features.

[0036] The present invention also provides a speech synthesis device, comprising:

[0037] An acquisition module is used to acquire the text to be synthesized and the timbre reference audio;

[0038] A decoding model module, used for inputting the text to be synthesized into a pre-trained decoding model, so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is trained using training text and corresponding training semantic features, the training semantic features are generated using training speech audio corresponding to the training text, and the training speech audio has a preset speech style;

[0039] The speech synthesis module is used to input the semantic features, the text to be synthesized and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio.

[0040] The present invention also provides an electronic device, comprising:

[0041] Memory for storing computer programs;

[0042] A processor is used to implement the speech synthesis method as described above when executing the computer program.

[0043] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the speech synthesis method as described above is implemented.

[0044] The present invention provides a speech synthesis method, comprising: obtaining a text to be synthesized and a timbre reference audio; inputting the text to be synthesized into a pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is trained using training text and corresponding training semantic features, the training semantic features are generated using training speech audio corresponding to the training text, and the training speech audio has a preset speech style; the semantic features, the text to be synthesized and the timbre reference audio are input into the pre-trained speech synthesis model for speech synthesis, and synthesized speech audio is obtained.

[0045] It can be seen that the present invention can first obtain the text to be synthesized and the timbre reference audio, wherein the text to be synthesized is used to provide text information, and the timbre reference audio is used to provide timbre information. Subsequently, the text to be synthesized can be input into the pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains the semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is trained using the training text and the corresponding training semantic features, the training semantic features are generated using the training speech audio corresponding to the training text, and the training speech audio has a preset speech style. In other words, the pre-trained decoding model can process the text to be synthesized to obtain semantic features, so that the semantic features contain both the text information of the text to be synthesized and the corresponding speech style information. Subsequently, the present invention can input the semantic features, the text to be synthesized and the timbre reference audio into the pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio. Considering that the semantic features contain speech style information, the present invention can input the speech style and the speech timbre separately into the pre-trained speech synthesis model, thereby realizing the decoupling of timbre style to improve the flexibility of speech synthesis. The present invention also provides a speech synthesis device, an electronic device and a computer-readable storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a pre-trained semantic model provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a pre-trained speech synthesis model provided by an embodiment of the present invention;

[0050] Figure 4 A structural block diagram of a speech synthesis device provided by an embodiment of the present invention;

[0051] Figure 5 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Speech synthesis technology refers to the technology of converting text content into speech audio. With the continuous development of artificial intelligence technology, especially with the continuous advancement of large-scale speech synthesis models, the effect of speech synthesis is improving day by day. However, in related technologies, although the speech synthesis model can directly convert text content into speech audio with a specific timbre, it cannot independently adjust the speech style of the speech audio generated by the model, thereby reducing the flexibility of speech synthesis.

[0054] In view of this, in order to solve the technical problem of how to decouple speech timbre and speech style in speech synthesis to improve the flexibility of speech synthesis, the present invention can provide a speech synthesis method, which can use semantic features (semantic tokens) including speech content and speech style to perform speech synthesis together with the text to be synthesized and the timbre reference audio to achieve the decoupling of speech timbre and style.

[0055] It should be noted that the present embodiment does not limit the hardware device for executing the present method, and can be set according to actual application requirements, for example, it can be a personal computer, a mobile device (such as a mobile phone, a tablet computer), a server, etc. If the present method is executed by a server, the server can also interact with a user terminal (such as a personal computer, a mobile device), such as the user terminal sending the text to be synthesized and the timbre reference audio to the server, and the server can send the synthesized speech audio to the user terminal after the complete audio synthesis.

[0056] For easier understanding, please refer to Figure 1 , Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present invention, the method may include:

[0057] S101, obtaining text to be synthesized and timbre reference audio.

[0058] In this step, the text to be synthesized is used to provide text content, so as to generate corresponding synthesized speech audio according to the text content. For example, the text to be synthesized can be text to be broadcast, lyrics, etc. The timbre reference audio is used to provide timbre information, so as to generate synthesized speech audio with corresponding timbre according to the timbre information. Among them, timbre is the sound feature of human voice, and the voices of different speakers can be distinguished by timbre.

[0059] S102. Input the text to be synthesized into a pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is trained using training text and corresponding training semantic features, the training semantic features are generated using training speech audio corresponding to the training text, and the training speech audio has a preset speech style.

[0060] Unlike related technologies, this embodiment does not directly use the text to be synthesized and the timbre reference audio for speech synthesis, but first decodes the text to be synthesized into corresponding semantic features, and then uses the semantic features, the text to be synthesized and the timbre reference audio for speech synthesis. Among them, the semantic features (semantic tokens) contain the speech content information and speech style information of the text to be synthesized. The speech content information is the text content of the text to be synthesized, and the speech style information is the speech style such as pronunciation and rhythm. It can be seen that the semantic features and the timbre reference audio contain the speech style information and the speech timbre information respectively, which can realize the decoupling and separation of the speech style and the speech timbre, thereby realizing the separate control of the speech style and the speech timbre.

[0061] In order to generate the above-mentioned semantic token, the present embodiment specially sets up a pre-trained decoding model. Specifically, in this step, the text to be synthesized can be input into the pre-trained decoding model, so that the pre-trained decoding model can determine the speech content and speech style corresponding to the text to be synthesized, thereby converting the text to be synthesized into the corresponding semantic features. Among them, the pre-trained decoding model is specifically trained using the training text and the corresponding training semantic features, and the training semantic features are generated using the training speech audio corresponding to the training text, and the training speech audio has a preset speech style. That is, after training, the pre-trained decoding model can understand how to give a speech style to the text content to ensure that the semantic features contain both the speech content and the speech style of the text to be synthesized.

[0062] Specifically, the pre-trained decoding model in this embodiment can be an autoregressive decoding model. Among them, the autoregressive decoding model is a neural network model that can automatically predict the next component in the sequence by measuring the previous input in the sequence. In this embodiment, the pre-trained decoding model is used to predict multiple semantic features based on the text to be synthesized, so as to provide the speech content information and speech style information required for generating a synthetic speech audio of a preset duration through these multiple semantic features. That is, the number of semantic features is related to the duration of the synthetic speech audio, and the numerical relationship between the two can be a multiple relationship of the audio sampling rate. In the process of generating semantic features, since the autoregressive decoding model needs to predict the next component in the sequence based on the previous input, the text to be synthesized needs to be input into the pre-trained decoding model to obtain the first semantic feature during the first reasoning, and in subsequent reasoning, the text to be synthesized and all previously generated semantic features need to be input into the pre-trained decoding model to obtain the next semantic feature.

[0063] Based on this, the pre-trained decoding model is an autoregressive decoding model; the inputting of the text to be synthesized into the pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains the semantic features corresponding to the text to be synthesized, may include:

[0064] Step 11: Input the text to be synthesized into the pre-trained decoding model to obtain the first semantic feature.

[0065] Step 12: Determine whether the number of semantic features currently generated reaches a preset value; wherein the preset value is set according to a preset duration of the synthesized speech audio; if the number of semantic features does not reach the preset value, proceed to step 13; if the number of semantic features reaches the preset value, proceed to step 14.

[0066] Step 13: Input the text to be synthesized and all the semantic features currently generated into the pre-trained decoding model to obtain the next semantic feature, and proceed to step 12.

[0067] Step 14: If yes, stop generating semantic features.

[0068] It should be noted that this embodiment does not limit the specific value of the above-mentioned preset value and the preset duration of the synthesized speech audio associated with this preset value, and can be set according to actual application requirements.

[0069] S103, inputting the semantic features, the text to be synthesized and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio.

[0070] In this step, the semantic features, the text to be synthesized, and the timbre reference audio can be input into the pre-trained speech synthesis model for speech synthesis, so that the speech style information and the timbre information can be input into the pre-trained speech synthesis model separately, thereby achieving the decoupling of the speech style and the speech timbre. In this way, when the speech style needs to be adjusted, only the pre-trained decoding model and the semantic features need to be adjusted, so that the speech style can be independently controlled.

[0071] It should be noted that the pre-trained speech synthesis model is a neural network model for generating synthesized speech audio according to a given input (such as speech text), and has been trained. In order to improve the speech synthesis effect, the pre-trained speech synthesis model in this embodiment may include an attention module, a timbre encoder and a decoder. Among them, the attention module is used to perform attention processing on the semantic features and the text to be synthesized to enhance the text information in the semantic features. The timbre encoder is used to convert the linear spectrum corresponding to the timbre reference audio into timbre representation information. The decoder is the core module of the pre-trained speech synthesis model, which is used to generate the corresponding synthesized speech audio according to the enhanced semantic features and timbre representation information. It should be pointed out that in the related art, the decoder usually receives the text to be synthesized and the timbre representation information to synthesize the synthesized speech audio with specified content and specified timbre. In the present invention, the decoder receives the enhanced semantic features and timbre representation information, and can synthesize the synthesized speech audio with specified content, specified style and specified timbre.

[0072] Based on this, the step of inputting the semantic features, the text to be synthesized, and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio may include:

[0073] Step 21: Input the semantic features and the text to be synthesized into the attention module, so that the attention module uses the text to be synthesized to perform attention processing on the semantic features to obtain enhanced semantic features.

[0074] In this step, in order to enhance the text content in the semantic feature, the semantic feature can be set as a query parameter (query), and the text to be synthesized can be set as a key parameter (Key) and a value parameter (Value), and then the attention mechanism is processed according to the query parameter, key parameter and value parameter. Among them, the query parameter, key parameter and value parameter are the core parameters in the attention mechanism (Attention).

[0075] Step 22: Convert the timbre reference audio into a linear spectrum, and input the linear spectrum into the timbre encoder to obtain timbre representation information.

[0076] In this step, the linear spectrum is obtained by performing a short-time Fourier transform on the timbre reference audio. The timbre encoder can take the linear spectrum as input and extract the timbre information in the timbre reference audio to control the timbre of the synthesized speech. The timbre representation information is the high-dimensional feature information of the timbre.

[0077] Step 23: Input the enhanced semantic features and the timbre representation information into the decoder to obtain the synthesized speech audio.

[0078] In this step, the decoder can use the VITS model (Variational Inference with adversarial learning for end-to-end Text-to-Speech, used for speech synthesis with adversarial learning conditional variational autoencoder) as the main structure. The difference from the VITS model is that: 1. The decoder in this embodiment takes the enhanced semantic features as input instead of the text to be synthesized as input; 2. The decoder in this embodiment does not need to set a duration prediction unit because the number of semantic features already indicates the audio duration of the synthesized speech audio.

[0079] Based on the above embodiments, the present invention can first obtain the text to be synthesized and the timbre reference audio, wherein the text to be synthesized is used to provide text information, and the timbre reference audio is used to provide timbre information. Subsequently, the text to be synthesized can be input into the pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains the semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is trained using the training text and the corresponding training semantic features, the training semantic features are generated using the training speech audio corresponding to the training text, and the training speech audio has a preset speech style. In other words, the pre-trained decoding model can process the text to be synthesized to obtain semantic features, so that the semantic features contain both the text information of the text to be synthesized and the corresponding speech style information. Subsequently, the present invention can input the semantic features, the text to be synthesized and the timbre reference audio into the pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio. Considering that the semantic features contain speech style information, the present invention can input the speech style and the speech timbre separately into the pre-trained speech synthesis model, thereby realizing the decoupling of timbre style to improve the flexibility of speech synthesis.

[0080] Based on the above embodiment, the training process of the pre-trained decoding model is introduced below. In one possible case, the training process of the pre-trained decoding model may include:

[0081] S201. Obtain the training text and the training speech audio.

[0082] In this step, the training text and the training voice audio appear in pairs, and the training voice audio has a preset voice style, which can be manually set and classified. It should be noted that this embodiment does not limit the number of training texts and training voice audio, which can be based on actual application requirements.

[0083] S202: extracting features from the training speech audio to obtain audio features, and quantizing the audio features to obtain a training semantic feature sequence.

[0084] In this step, in order to pre-construct a training semantic feature sequence, a pre-trained audio feature extraction model can be used to extract features from the training speech audio to obtain audio features, and the audio features can be quantized to obtain a training semantic feature sequence. Among them, the quantization process is used to reduce the dimensionality of high-dimensional audio features so that they can be processed by pre-training decoding models and pre-training speech synthesis models. Among them, the pre-trained audio feature extraction model can be set according to actual application requirements, such as HuBERT (Hidden unit BERT, BERT model with hidden units. BERT, Bidirectional Encoder Representations from Transformers, a pre-trained language model based on a bidirectional Transformer encoder), and the extracted audio features can be hidden features of the training speech audio. The above-mentioned quantization method can also be set according to actual application requirements. It is only necessary to map continuous high-dimensional audio features from the real number domain to integers to convert high-dimensional audio features into a finite number of discrete values.

[0085] S203: training the initial decoding model using the training text and the training semantic feature sequence to obtain the pre-trained decoding model.

[0086] Specifically, the training process of the initial decoding model using the training text and the training semantic feature sequence to obtain the pre-trained decoding model may include:

[0087] Step 31: input the training text into the initial decoding model to obtain a semantic feature sequence to be tested.

[0088] Step 32: using a preset loss function to determine the loss value between the training semantic feature sequence and the semantic feature sequence to be tested.

[0089] Step 33: Use the loss value to update the parameters of the initial decoding model to obtain the pre-trained decoding model.

[0090] In steps 31 to 33, the training text can be input into the initial decoding model to obtain the semantic feature sequence to be tested, and the loss value between the training semantic feature sequence and the semantic feature sequence to be tested can be determined using a preset loss function, and then the initial decoding model can be updated using the loss value to obtain the required pre-trained decoding model. The initial decoding model refers to an untrained pre-trained decoding model. Since the model usually needs to be trained multiple times before it can converge (that is, the loss value is less than a preset threshold), the above steps 31 to 33 can be performed multiple times until the loss value between the semantic feature sequence to be tested generated by the initial decoding model and the given training semantic feature sequence is less than a preset threshold.

[0091] It should be noted that this embodiment does not limit the specific preset loss function, which can be set according to actual application requirements, for example, it can be a cross entropy loss function.

[0092] Based on the above embodiment, the training process of the pre-trained speech synthesis model is introduced below. In one possible case, the training process of the pre-trained speech synthesis model may include:

[0093] S301, obtaining the training text, the training speech audio, the training semantic features and the training timbre reference audio.

[0094] In this step, the training text, training speech audio, training semantic features, and training timbre reference audio appear in pairs, where the training text, training semantic features, and training timbre reference audio will be used as model input, and the training speech audio will be used to verify the speech synthesis effect of the speech synthesis model. This embodiment does not limit the number of training texts, training speech audios, training semantic features, and training timbre reference audios, which can be based on actual application requirements.

[0095] Furthermore, the pre-trained speech synthesis model can be trained before the pre-trained decoding model, or after the pre-trained decoding model. When the pre-trained speech synthesis model can be trained before the pre-trained decoding model, a pre-trained audio feature extraction model can be used to extract features from the training speech audio to obtain audio features, and then the audio features can be quantized to obtain training semantic features. The pre-trained audio feature extraction model can be HuBERT.

[0096] Based on this, the acquisition of training semantic features may include:

[0097] Step 41: extracting features from the training speech audio using a pre-trained audio feature extraction model to obtain audio features;

[0098] Step 42: quantize the audio features to obtain the training semantic features.

[0099] S302: Input the training semantic features and the training text into the attention module, so that the attention module performs attention processing on the training semantic features using the training text to obtain training enhanced semantic features.

[0100] S303: Convert the training timbre reference audio into a linear spectrum, and input the linear spectrum into the timbre encoder to obtain training timbre representation information.

[0101] S304: Input the training enhanced semantic features and the training timbre representation information into the decoder to obtain the synthesized speech audio to be tested.

[0102] S305. Calculate a loss value using the training speech audio and the synthesized speech audio to be tested, and use the loss value to update the attention module, the timbre encoder and the decoder to obtain the pre-trained speech synthesis model.

[0103] Similar to the pre-trained decoding model, the speech synthesis model usually needs to be trained multiple times before it can converge (that is, the loss value is less than a preset threshold). Therefore, the above steps S302~S305 can be executed multiple times until the loss value between the training speech audio and the synthesized speech audio to be tested is less than a preset threshold.

[0104] In addition, the decoder can also be trained in a separate training method. Specifically, in the decoder, the audio representation information z (audio hidden features) can first be modeled to obtain the posterior distribution q(z), and the Gaussian distribution can be used to approximate q(z). In addition, the feature x of the semantic feature can be passed through another encoder (VAE) to obtain the conditional prior distribution p(z|x). Since the distribution of p(z|x) is unknown, in order to maintain its modeling ability, p(z|x) can be constrained to a Gaussian distribution instead of being constrained to a Gaussian distribution. Instead, a regularized flow model is used to imply p(z|x) to imply it to the prior distribution p(z). Here, it is assumed that p(z) is a Gaussian distribution, and then the KL loss function (KL divergence / relative entropy) is used to make the two distributions of p(z) and q(z) close, so as to update the decoder parameters.

[0105] Furthermore, in order to improve the quality of synthesized audio, the present embodiment may also introduce a pre-trained discriminator for adversarial training. The pre-trained discriminator is a binary classification model, which is trained using real audio and synthesized audio and can effectively distinguish real audio from synthesized audio. The pre-trained discriminator can be used to classify the synthesized speech audio generated by the decoder, and the classification result can be converted into a loss value to continue to update the parameters of the decoder.

[0106] Based on the above embodiment, the above speech synthesis method is introduced below according to a specific schematic diagram. Figure 2 , Figure 2A schematic diagram of a pre-trained semantic model provided in an embodiment of the present invention. The model inputs text and outputs semantic features corresponding to the text. During prediction, autoregressive decoding is required, and the specific operation is that each decoding step of the model will accumulate the previous output as input. Therefore, the semantic features input in the schematic diagram are the semantic features previously predicted by the model. During training, the pre-trained self-supervised speech model Hubert can be used to extract hidden features from the audio, and the hidden features can be vectorized to obtain semantic features, so that the autoregressive semantic model in this embodiment can be trained. During prediction, given the text, the autoregressive semantic model outputs the semantic features corresponding to the text in the form of autoregressive decoding, and the semantic features contain information such as the content, duration and rhythm of the speech to be synthesized.

[0107] Please refer to Figure 3 , Figure 3 The schematic diagram of a pre-trained speech synthesis model provided by an embodiment of the present invention. The pre-trained speech synthesis model is mainly composed of three submodules, namely, an attention module, a timbre encoder and a decoder module. First, the semantic features and text will pass through the attention module, and output a feature sequence of the same length as the semantic features. This step is to enhance the text information in the semantic features so that the intelligibility of the synthesized content is guaranteed. The timbre encoder accepts the linear spectrum as input, extracts the timbre information in the speech to control the timbre of the synthesized speech, and the linear spectrum is obtained from the speech through short-time Fourier transform. Finally, the semantic features and timbre features are sent to the decoder module together to synthesize the speech waveform we need. The structure of the decoder is the most critical. The decoder of this embodiment uses VITS as a main structure. This embodiment modifies the original input of VITS. Originally, VITS accepts text as input, but in this embodiment, it is changed to accept semantic features as input. The text and semantic features are not equal in time sequence. The number of semantic features depends on the duration of the synthesized speech audio, which is a multiple of the sampling rate.

[0108] The following is the training process of the above two models:

[0109] 1. First, prepare the data. This embodiment requires thousands or even tens of thousands of hours of training data. This is because the number of model parameters to be trained is larger than that of previous speech synthesis models, so the amount of data is also required.

[0110] 2. After the data is prepared, the speech synthesis model is trained first. All three modules (attention module, timbre encoder, and decoder) will participate in the training. In addition, the decoder will add a separate quantization module during the training process, which will only be used during the training process. The input of the audio reconstruction model is text and audio self-supervision features. The audio self-supervision features are extracted from Hubert. The audio self-supervision features will be discretely encoded into semantic features through the quantization module, and then the semantic features will be sent to the vits module in the decoder to synthesize audio. VITs is mainly composed of two parts. On the one hand, it is the self-reconstruction of audio. The audio is encoded by the variational autoencoder (VAE) to obtain the audio hidden feature z, and the posterior distribution q(z) of z is modeled. The Gaussian distribution is used to approximate q(z); on the other hand, the semantic feature x is passed through another VAE to obtain the conditional prior distribution p(z|x). Since the distribution of p(z|x) is unknown, in order to maintain its modeling ability, this embodiment does not constrain p(z|x) to be a Gaussian distribution, but uses a regularized flow model to imply p(z|x), which is implied as the previous prior distribution p(z). Here, it is assumed that p(z) is a Gaussian distribution, and then the two distributions of p(z) and q(z) are made close by KLloss. In addition, the Mel reconstruction loss is calculated between the audio synthesized by the decoder and the real audio to ensure the effectiveness of VAE training. Furthermore, in order to improve the quality of the synthesized audio, a discriminator is added, adversarial training is introduced, the authenticity of the synthesized audio is identified, the effect of the synthesized audio is improved, and finally a trained pre-trained speech synthesis model is obtained.

[0111] 3. After training the audio reconstruction model, you can call the quantization module to quantize the audio self-supervision features and obtain the semantic features corresponding to each sentence. In this way, the training data, text and corresponding semantic features of the autoregressive decoding model are prepared. Next, start training the autoregressive decoding model. First, construct the input and output, that is, input text and output semantic features. The predicted semantic features at each step will calculate the cross entropy with the real semantic features to constrain the accuracy of the model's prediction classification. In this way, the autoregressive semantic model can be trained. When predicting, autoregressive decoding is required. The specific operation is that the model will accumulate the previous output as input at each decoding step.

[0112] The following is an introduction to the speech synthesis device, electronic device and computer-readable storage medium provided in the embodiments of the present invention. The speech synthesis device, electronic device and computer-readable storage medium described below can be referenced to the speech synthesis method described above.

[0113] Please refer to Figure 4 , Figure 4 A structural block diagram of a speech synthesis device provided by an embodiment of the present invention, the device may include:

[0114] An acquisition module 401 is used to acquire the text to be synthesized and the timbre reference audio;

[0115] The decoding model module 402 is used to input the text to be synthesized into a pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is trained using the training text and the corresponding training semantic features, the training semantic features are generated using the training speech audio corresponding to the training text, and the training speech audio has a preset speech style;

[0116] The speech synthesis module 403 is used to input the semantic features, the text to be synthesized and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio.

[0117] Optionally, the pre-trained decoding model is an autoregressive decoding model; the decoding model module 402 includes:

[0118] A first input submodule, used for inputting the text to be synthesized into the pre-trained decoding model to obtain a first semantic feature;

[0119] A judgment submodule, used to judge whether the number of semantic features currently generated has reached a preset value; wherein the preset value is set according to a preset duration of the synthesized speech audio;

[0120] A second input submodule is used for, if not, inputting the text to be synthesized and all the semantic features currently generated into the pre-trained decoding model to obtain the next semantic feature, and entering the step of determining whether the number of semantic features currently generated reaches a preset value;

[0121] The stop submodule is used to stop generating semantic features if .

[0122] Optionally, the device may further include:

[0123] A first acquisition module, used to acquire the training text and the training voice audio;

[0124] A training semantic feature sequence generation module is used to extract features from the training speech audio to obtain audio features, and to quantize the audio features to obtain a training semantic feature sequence;

[0125] The first training module is used to train the initial decoding model using the training text and the training semantic feature sequence to obtain the pre-trained decoding model.

[0126] Optionally, the first training module includes:

[0127] A processing submodule, used for inputting the training text into the initial decoding model to obtain a semantic feature sequence to be tested;

[0128] A loss value calculation submodule, used to determine the loss value between the training semantic feature sequence and the semantic feature sequence to be tested using a preset loss function;

[0129] An updating submodule is used to update the parameters of the initial decoding model using the loss value to obtain the pre-trained decoding model.

[0130] Optionally, the pre-trained speech synthesis model includes an attention module, a timbre encoder and a decoder; the speech synthesis module 403 includes:

[0131] An attention processing submodule, used for inputting the semantic features and the text to be synthesized into the attention module, so that the attention module performs attention processing on the semantic features using the text to be synthesized to obtain enhanced semantic features;

[0132] A timbre extraction submodule, used for converting the timbre reference audio into a linear spectrum, and inputting the linear spectrum into the timbre encoder to obtain timbre representation information;

[0133] The speech synthesis submodule is used to input the enhanced semantic features and the timbre representation information into the decoder to obtain the synthesized speech audio.

[0134] Optionally, the device may further include:

[0135] A second acquisition module is used to acquire the training text, the training speech audio, the training semantic features and the training timbre reference audio;

[0136] A first processing module, configured to input the training semantic features and the training text into the attention module, so that the attention module performs attention processing on the training semantic features using the training text to obtain training enhanced semantic features;

[0137] A second processing module is used to convert the training timbre reference audio into a linear spectrum, and input the linear spectrum into the timbre encoder to obtain training timbre representation information;

[0138] A third processing module, used for inputting the training enhanced semantic features and the training timbre representation information into the decoder to obtain the synthesized speech audio to be tested;

[0139] An updating module is used to calculate a loss value using the training speech audio and the synthesized speech audio to be tested, and to use the loss value to update the attention module, the timbre encoder and the decoder to obtain the pre-trained speech synthesis model.

[0140] Optionally, the second acquisition module may include:

[0141] An audio feature extraction submodule, used to extract features from the training speech audio using a pre-trained audio feature extraction model to obtain audio features;

[0142] The training semantic feature generation submodule is used to quantize the audio features to obtain the training semantic features.

[0143] Please refer to Figure 5 , Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention provides an electronic device 10, including a processor 11 and a memory 12; wherein the memory 12 is used to store a computer program; and the processor 11 is used to execute the speech synthesis method provided in the aforementioned embodiment when executing the computer program.

[0144] For the specific process of the above-mentioned speech synthesis method, please refer to the corresponding content provided in the above-mentioned embodiments, which will not be repeated here.

[0145] Furthermore, the memory 12 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0146] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16; wherein the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0147] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the speech synthesis method described in the above embodiment when executed by a processor.

[0148] Since the embodiments of the computer program product part correspond to the embodiments of the speech synthesis method part, please refer to the description of the embodiments of the speech synthesis method part for the embodiments of the computer program product part, and will not be repeated here.

[0149] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method described in the above embodiment is implemented.

[0150] Since the embodiments of the computer-readable storage medium part and the embodiments of the speech synthesis method part correspond to each other, the embodiments of the storage medium part refer to the description of the embodiments of the speech synthesis method part, which will not be repeated here.

[0151] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0152] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0153] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0154] The above is a detailed introduction to a speech synthesis method, device, electronic device and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Obtain the text to be synthesized and the timbre reference audio; Input the text to be synthesized into a pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is an autoregressive decoding model, the pre-trained decoding model is trained using training text and corresponding training semantic features, the training semantic features are generated using training speech audio corresponding to the training text, and the training speech audio has a preset speech style; Inputting the semantic features, the text to be synthesized and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio; The step of inputting the text to be synthesized into a pre-trained decoding model so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized, including: Inputting the text to be synthesized into the pre-trained decoding model to obtain a first semantic feature; Determine whether the number of semantic features currently generated reaches a preset value; wherein the preset value is set according to a preset duration of the synthesized speech audio; If not, the text to be synthesized and all the semantic features currently generated are input into the pre-trained decoding model to obtain the next semantic feature, and then the step of determining whether the number of semantic features currently generated reaches a preset value is entered; If yes, stop generating the semantic feature.

2. The speech synthesis method according to claim 1, characterized in that: The training process of the pre-trained decoding model includes: Acquire the training text and the training voice audio; Extracting features from the training speech audio to obtain audio features, and quantizing the audio features to obtain a training semantic feature sequence; The initial decoding model is trained using the training text and the training semantic feature sequence to obtain the pre-trained decoding model.

3. The speech synthesis method according to claim 2, characterized in that: The method of training the initial decoding model using the training text and the training semantic feature sequence to obtain the pre-trained decoding model includes: Inputting the training text into the initial decoding model to obtain a semantic feature sequence to be tested; Determining the loss value between the training semantic feature sequence and the semantic feature sequence to be tested by using a preset loss function; The loss value is used to update the parameters of the initial decoding model to obtain the pre-trained decoding model.

4. The speech synthesis method according to any one of claims 1 to 3, characterized in that: The pre-trained speech synthesis model includes an attention module, a timbre encoder and a decoder; The step of inputting the semantic features, the text to be synthesized and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio comprises: Inputting the semantic feature and the text to be synthesized into the attention module, so that the attention module performs attention processing on the semantic feature using the text to be synthesized to obtain an enhanced semantic feature; Converting the timbre reference audio into a linear spectrum, and inputting the linear spectrum into the timbre encoder to obtain timbre representation information; The enhanced semantic features and the timbre representation information are input into the decoder to obtain the synthesized speech audio.

5. The speech synthesis method according to claim 4, characterized in that: The training process of the pre-trained speech synthesis model includes: Acquire the training text, the training speech audio, the training semantic features and the training timbre reference audio; Inputting the training semantic features and the training text into the attention module, so that the attention module performs attention processing on the training semantic features using the training text to obtain training enhanced semantic features; Converting the training timbre reference audio into a linear spectrum, and inputting the linear spectrum into the timbre encoder to obtain training timbre representation information; Inputting the training enhanced semantic features and the training timbre representation information into the decoder to obtain the synthesized speech audio to be tested; The loss value is calculated using the training speech audio and the synthesized speech audio to be tested, and the attention module, the timbre encoder and the decoder are updated using the loss value to obtain the pre-trained speech synthesis model.

6. The speech synthesis method according to claim 5, characterized in that: Get training semantic features, including: Using a pre-trained audio feature extraction model to extract features from the training speech audio to obtain audio features; The audio features are quantized to obtain the training semantic features.

7. A speech synthesis device, characterized in that: include: An acquisition module is used to acquire the text to be synthesized and the timbre reference audio; A decoding model module, used for inputting the text to be synthesized into a pre-trained decoding model, so that the pre-trained decoding model determines the speech content and speech style corresponding to the text to be synthesized, and obtains semantic features corresponding to the text to be synthesized; wherein the pre-trained decoding model is an autoregressive decoding model, the pre-trained decoding model is trained using training text and corresponding training semantic features, the training semantic features are generated using training speech audio corresponding to the training text, and the training speech audio has a preset speech style; A speech synthesis module, used for inputting the semantic features, the text to be synthesized and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain synthesized speech audio; The decoding model module comprises: A first input submodule, used for inputting the text to be synthesized into the pre-trained decoding model to obtain a first semantic feature; A judgment submodule, used to judge whether the number of semantic features currently generated has reached a preset value; wherein the preset value is set according to a preset duration of the synthesized speech audio; A second input submodule is used for, if not, inputting the text to be synthesized and all the semantic features currently generated into the pre-trained decoding model to obtain the next semantic feature, and entering the step of determining whether the number of semantic features currently generated reaches a preset value; The stopping submodule is used for stopping generating the semantic feature if the condition is true.

8. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the speech synthesis method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.

Images