Speech synthesis method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a speech synthesis method and device, equipment and a storage medium. The method comprises the following steps: receiving a target text of speech to be synthesized; inputting a target phoneme sequence corresponding to the target text into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is obtained by training a posterior encoder, a bypass guide module, a training record and a training text; the bypass guide module is used for generating guide information; the guiding information is used for guiding the posterior encoder, so that the posterior encoder extracts backbone voice information in a linear spectrum corresponding to the training record, and the backbone voice information is used for training the voice synthesis model. According to the embodiment of the invention, the overall training effect of the speech synthesis model is improved, the problems of inaccurate pronunciation and tone quality fluctuation of the target speech synthesized by the model are effectively solved, and the clarity and naturalness of the target speech are enhanced.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method and device, an electronic device and a storage medium. Background Art

[0002] During the training process, existing speech synthesis models usually use a conventional convolutional neural network with multiple layers connected in series as a posterior encoder. Although this design can capture the statistical characteristics of speech signals to a certain extent, due to its structural limitations, it lacks clear directionality and specificity when extracting backbone information of speech. Specifically, when processing speech data, the transformation direction of conventional convolutional neural networks is random, and it cannot focus on the core features of speech signals in a targeted manner, but may be disturbed by unimportant information. This random and lack of directionality in feature extraction makes it difficult for the model to stably capture the key information of speech signals during training, thus affecting the effect of the overall speech synthesis model.

[0003] In practical applications, this unstable feature extraction method may lead to problems such as inaccurate pronunciation and fluctuating sound quality in the synthesized speech. For example, in some cases, the model may over-focus on minor features in the speech signal and ignore the key parts of the pronunciation, resulting in a decrease in the clarity and naturalness of the synthesized speech, affecting the user experience. Summary of the invention

[0004] The embodiments of the present application provide a speech synthesis method to solve or at least partially solve the above problems.

[0005] Correspondingly, the embodiments of the present application also provide a speech synthesis device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0006] In order to solve the above problems, the present application discloses a speech synthesis method, which includes:

[0007] Receiving target text for speech synthesis;

[0008] The target phoneme sequence corresponding to the target text is input into a speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model.

[0009] Optionally, the bypass guidance module is composed of a plurality of coupling layers connected in series, and the training process of the speech synthesis model includes:

[0010] Inputting pronunciation information corresponding to the training recording into the bypass guidance module;

[0011] When the pronunciation information passes through the coupling layers in the bypass guidance module one by one, the guidance information output by each coupling layer is obtained, and the self-supervision loss output by the last coupling layer is obtained;

[0012] Inputting the linear spectrum corresponding to the guide information and the training recording into the posterior encoder to obtain the training speech mean and the training speech variance;

[0013] The speech synthesis model is trained according to the training speech mean, the training speech variance, the self-supervision loss and the training text.

[0014] Optionally, the a posteriori encoder is composed of a plurality of convolutional layers connected in series, the number of the convolutional layers is the same as the number of coupling layers in the bypass guidance module, each of the convolutional layers corresponds to each of the coupling layers one by one, and inputting the linear spectrum corresponding to the guidance information and the training recording into the a posteriori encoder comprises:

[0015] If the current convolutional layer is the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the linear spectrum corresponding to the training recording are input into the current convolutional layer;

[0016] If the current convolutional layer is not the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the output information of the previous convolutional layer are input into the current convolutional layer.

[0017] Optionally, the method further comprises:

[0018] Generate a first guided coefficient corresponding to the current convolutional layer according to the guided information output by the coupling layer corresponding to the current convolutional layer;

[0019] Calculating a linear spectrum corresponding to the training recording, or a second guided coefficient corresponding to the output information of the previous convolutional layer;

[0020] Calculating according to the first guided coefficient and the second guided coefficient to obtain the output of the current convolutional layer;

[0021] If the current convolutional layer is the last convolutional layer in the posterior encoder, the training speech mean and the training speech variance are obtained according to the output of the current convolutional layer.

[0022] Optionally, the speech synthesis model further includes a decoder, a stream module and a text processing module, and the training of the speech synthesis model according to the training speech mean, the training speech variance, the self-supervision loss and the training text includes:

[0023] Randomly sampling the Gaussian distribution model corresponding to the training speech mean and the training speech variance to obtain a speech sampling result;

[0024] Inputting the speech sampling result into the decoder to obtain the speech generation loss;

[0025] Inputting the speech sampling result into the stream module to obtain a training intermediate vector;

[0026] Inputting the training intermediate vector and the training text into the text processing module to obtain the duration loss and relative entropy;

[0027] The neural network parameters of the speech synthesis model are updated according to the self-supervision loss, the speech generation loss, the duration loss and the relative entropy.

[0028] Optionally, the text processing module includes a text encoder, a projection layer, and a random duration predictor, and the inputting of the training intermediate vector and the training text into the text processing module to obtain the duration loss and the relative entropy includes:

[0029] Inputting the phoneme sequence corresponding to the training text into the text encoder to obtain a training text matrix;

[0030] Input the training text matrix into the projection layer to obtain the training text mean and the training text variance;

[0031] For the training text mean and the training text variance, the training intermediate vector is used to perform time series alignment to obtain a first training alignment parameter;

[0032] Inputting the training text matrix into the random duration predictor to obtain a second training alignment parameter;

[0033] Comparing the first training alignment parameter with the second training alignment parameter to obtain the duration loss;

[0034] The relative entropy is obtained by comparing the training text Gaussian distribution shaped by the training text mean and the training text variance with the training speech Gaussian distribution shaped by the training speech mean and the training speech variance.

[0035] Optionally, inputting a target phoneme sequence corresponding to the target text into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model comprises:

[0036] Inputting a target phoneme sequence corresponding to the target text into the text encoder to obtain a text matrix corresponding to the target phoneme sequence;

[0037] Input the text matrix into the projection layer to obtain the text mean and text variance;

[0038] Randomly sampling the text Gaussian distribution corresponding to the text mean and the text variance to obtain a text sampling result;

[0039] Inputting the text matrix into the random duration predictor to obtain alignment parameters;

[0040] According to the alignment parameters, the text sampling result is subjected to a time-to-time copy operation to obtain a target text sampling result;

[0041] Inputting the target text sampling result into the stream module to obtain an intermediate vector;

[0042] The intermediate vector is input into the decoder, and the target speech corresponding to the target text is output.

[0043] The present application also discloses a speech synthesis device, which includes:

[0044] A text receiving module, used for receiving a target text to be synthesized into speech;

[0045] A speech synthesis module is used to input the target phoneme sequence corresponding to the target text into a speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model.

[0046] The embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the speech synthesis method as described in one or more of the embodiments of the present application.

[0047] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the speech synthesis method as described in one or more of the embodiments of the present application.

[0048] Compared with the prior art, the embodiments of the present application have the following advantages:

[0049] In an embodiment of the present application, a target text of speech to be synthesized is received; a target phoneme sequence corresponding to the target text is input into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model. The embodiment of the present application configures a bypass guidance module for the posteriori encoder, thereby realizing the use of the guidance information output by the bypass guidance module to specifically guide the posteriori encoder to capture the backbone information in the linear spectrum corresponding to the training recording. The embodiment of the present application helps to improve the overall training effect of the speech synthesis model, effectively solves the problems of inaccurate pronunciation and fluctuating sound quality of the target speech synthesized by the model, and enhances the clarity and naturalness of the target speech, thereby significantly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flowchart of a speech synthesis method embodiment of the present application;

[0051] Figure 2 is a schematic diagram of a coupling layer of an embodiment of a speech synthesis method of the present application;

[0052] Figure 3 It is a normalization guidance schematic diagram of a speech synthesis method embodiment of the present application;

[0053] Figure 4 This is a schematic diagram of a speech synthesis model structure of a speech synthesis method embodiment of the present application;

[0054] Figure 5 is a structural block diagram of an embodiment of a speech synthesis device of the present application;

[0055] Figure 6 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0057] Reference Figure 1 is a flowchart of the steps of an embodiment of a speech synthesis method of the present application, including the following steps:

[0058] Step 101: Receive the target text of the speech to be synthesized.

[0059] In step 101, before using the speech synthesis model to synthesize the target speech, it is necessary to first receive the target text of the speech to be synthesized.

[0060] In one embodiment, a third-party phonetic annotation tool (such as pypinyin, a Python library for converting Chinese characters into pinyin) can be used first to translate the target text of the speech to be synthesized into a target phoneme sequence. In one example, the target phoneme sequence is the initials and finals for Chinese. For example, the target phoneme sequence of "good person" is "h3ao3r2en2".

[0061] Step 102: Input the target phoneme sequence corresponding to the target text into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training texts; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recordings, and the backbone speech information is used to train the speech synthesis model.

[0062] In step 102, the target phoneme sequence translated from the target text of the speech to be synthesized is input into the speech synthesis model, so as to obtain the target speech corresponding to the target text output by the speech synthesis model.

[0063] In this embodiment, before using the speech synthesis model, it is necessary to first train the speech synthesis model. Specifically, when training the speech synthesis model, a posterior encoder, a bypass guidance module, training recordings, and training texts are required for training.

[0064] Existing posterior encoders usually adopt a conventional convolutional neural network structure with multiple layers in series. This structure theoretically forms a statistical transformation, but its direction is random and lacks clear directivity. It does not specifically extract the backbone information in the speech and may deviate to unimportant information. This uncertainty leads to unstable performance of the overall trained speech synthesis model, and problems such as inaccurate pronunciation and fluctuating sound quality may occur in the synthesized speech.

[0065] This embodiment solves this problem by introducing a bypass guidance module. The bypass guidance module is used to generate guidance information, and the guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, so that the backbone speech information can be used to train the speech synthesis model. For example, using the guidance information to guide the posterior encoder can enable the posterior encoder to better capture the backbone speech information such as phoneme information, pitch information, prosody information, and spectral features in the linear spectrum that are most critical for speech synthesis, thereby making the training effect of the speech synthesis model more stable.

[0066] In one embodiment, the training recording can be input into the HuBERT model (Hidden-Unit BERT, a speech representation model based on self-supervised learning) to obtain an output result, and the output result is used as the pronunciation information corresponding to the training recording, so that the pronunciation information corresponding to the training recording is used as the input of the bypass guidance module, so that the bypass guidance module outputs guidance information. Since the output result of the HuBERT model can inherently reflect the core information in the training speech, using it as the input of the bypass guidance module can enable the output of the bypass guidance module to assist the posterior encoder to better capture the most critical backbone speech information in the linear spectrum for speech synthesis.

[0067] In an embodiment of the present application, a target text of speech to be synthesized is received; a target phoneme sequence corresponding to the target text is input into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model. The embodiment of the present application configures a bypass guidance module for the posteriori encoder, thereby realizing the use of the guidance information output by the bypass guidance module to specifically guide the posteriori encoder to capture the backbone information in the linear spectrum corresponding to the training recording. The embodiment of the present application helps to improve the overall training effect of the speech synthesis model, effectively solves the problems of inaccurate pronunciation and fluctuating sound quality of the target speech synthesized by the model, and enhances the clarity and naturalness of the target speech, thereby significantly improving the user experience.

[0068] In an optional embodiment of the present application, the bypass guidance module is composed of a plurality of coupling layers connected in series, and the training process of the speech synthesis model includes:

[0069] Inputting pronunciation information corresponding to the training recording into the bypass guidance module;

[0070] When the pronunciation information passes through the coupling layers in the bypass guidance module one by one, the guidance information output by each coupling layer is obtained, and the self-supervision loss output by the last coupling layer is obtained;

[0071] Inputting the linear spectrum corresponding to the guide information and the training recording into the posterior encoder to obtain the training speech mean and the training speech variance;

[0072] The speech synthesis model is trained according to the training speech mean, the training speech variance, the self-supervision loss and the training text.

[0073] In this embodiment, the process of using the bypass guidance module to train the speech synthesis model is further specifically demonstrated. The internal structure of the bypass guidance module in this embodiment will adopt a "normalized flow", that is, the bypass guidance module is composed of several coupling layers connected in series.

[0074] Reference Figure 2 , is a schematic diagram of the coupling layer of an embodiment of a speech synthesis method of the present application.

[0075] Figure 2 The specific operation of a coupling layer in the bypass guidance module is shown in FIG. The coupling layer structure can realize the forward transformation from x to z and the reverse transformation from z to x. Figure 2 Shown in FIG. 1 is the process of forward transformation from x to z used in the training process of the speech synthesis model.

[0076] Specifically, if the dimensions of signal x and signal z are both D, then by presetting a constant d (d is less than D), the D-dimensional signal can be divided into two parts, namely x1, ..., x2 in the upper part d and the lower part of x d+1 ,……,x D In the coupled layer transformation, the upper part remains unchanged and can be directly obtained by x1, ..., x d Copy to get z1, ..., z d .

[0077] The lower part performs an affine transformation, and the coefficients and bias of the affine transformation are calculated by the upper part. Figure 2 As shown, β d+1 , ..., β D is the coefficient of the affine transformation, which is calculated by the neural network submodule F; d+1 ,……,γ D is the bias of the affine transformation, calculated by the neural network submodule H. d+1 ,……,x D After affine transformation, we can get z d+1 ,……,zD .

[0078] Therefore, each time a coupling layer transformation is passed, only a part of the dimension will change, and the rest of the dimensions remain unchanged. After each coupling layer transformation, a dimension flipping operation is introduced to swap the changed part with the unchanged part before entering the next coupling layer. In this way, the coupling layer transformations of an even number of layers in series can make the two dimensions undergo roughly the same number of transformations.

[0079] In this embodiment, the training process of the speech synthesis model can be specifically expressed as follows.

[0080] First, the pronunciation information corresponding to the training recording is input into the bypass guidance module. In one embodiment, the training recording can be input into the HuBERT model to obtain an output result, and the output result is used as the pronunciation information corresponding to the training recording, so that the pronunciation information corresponding to the training recording is used as the input of the bypass guidance module.

[0081] Next, when the pronunciation information passes through the coupling layers in the bypass guidance module one by one, the guidance information output by each coupling layer is obtained, and the self-supervision loss output by the last coupling layer is obtained. Specifically, the output of each coupling layer can be drawn out in the middle of the bypass guidance module, and finally these outputs drawn out from each coupling layer are summarized together to form a sequence of length N LAYERS as guidance information. Where N LAYERS is the number of coupling layers connected in series in the internal structure of the bypass guidance module. At the end of the bypass guidance module, the self-supervision loss can be output.

[0082] In one embodiment, the a posteriori encoder is composed of a plurality of convolutional layers connected in series, the number of convolutional layers is the same as the number of coupling layers in the bypass guidance module, and each convolutional layer corresponds to each coupling layer one by one. Therefore, the output of each coupling layer in the bypass guidance module is actually the guidance information required by each convolutional layer in the a posteriori encoder.

[0083] Afterwards, the linear spectrum corresponding to the guidance information and the training recording is input into the a posteriori encoder to obtain the training speech mean and the training speech variance. In one embodiment, the linear spectrum can be used as the main input, and the guidance information can be used as the bypass input, and both paths are sent to the a posteriori encoder to obtain the training speech mean and the training speech variance. These training speech means and training speech variances can shape the corresponding Gaussian distribution model.

[0084] The speech synthesis model is trained based on the training speech mean, training speech variance, self-supervised loss, and training text.

[0085] The embodiment of the present application configures a bypass guidance module for the a posteriori encoder and utilizes the guidance information output by the bypass guidance module to specifically guide the a posteriori encoder to capture the backbone information in the linear spectrum corresponding to the training recording.

[0086] In an optional embodiment of the present application, the a posteriori encoder is composed of a plurality of convolutional layers connected in series, the number of the convolutional layers is the same as the number of coupling layers in the bypass guidance module, each of the convolutional layers corresponds to each of the coupling layers one by one, and the step of inputting the linear spectrum corresponding to the guidance information and the training recording into the a posteriori encoder includes:

[0087] If the current convolutional layer is the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the linear spectrum corresponding to the training recording are input into the current convolutional layer;

[0088] If the current convolutional layer is not the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the output information of the previous convolutional layer are input into the current convolutional layer.

[0089] In this embodiment, the a posteriori encoder is composed of a number of convolutional layers connected in series, the number of convolutional layers is the same as the number of coupling layers in the bypass guidance module, and each convolutional layer corresponds to each coupling layer one by one, that is, the output of each coupling layer in the bypass guidance module is actually the guidance information required by each convolutional layer in the a posteriori encoder. In an example, the number of coupled layers connected in series in the internal structure of the bypass guidance module is also N LAYERS, and the number of convolutional layers connected in series in the internal structure of the a posteriori encoder is NLAYERS.

[0090] Specifically, the guidance information output by the coupling layer can normalize and guide the output of the convolution layer inside the convolution layer. If the current convolution layer is the first convolution layer in the structure of the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolution layer and the linear spectrum corresponding to the training recording are input into the current convolution layer. In one example, the current convolution layer is the first convolution layer in the structure of the posterior encoder, and the coupling layer corresponding to the current convolution layer is also the first coupling layer in the bypass guidance module. From the guidance information, the guidance information output by the first coupling layer in the bypass guidance module is obtained, and the guidance information output by the first coupling layer and the linear spectrum corresponding to the training recording are input into the current convolution layer.

[0091] If the current convolutional layer is not the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the output information of the previous convolutional layer are input into the current convolutional layer. In one example, if the current convolutional layer is the second convolutional layer in the structure of the posterior encoder, the coupling layer corresponding to the current convolutional layer is also the second coupling layer in the bypass guidance module. From the guidance information, the guidance information output by the second coupling layer in the bypass guidance module is obtained, and the guidance information output by the second coupling layer is input into the second convolutional layer together with the output information output by the first convolutional layer.

[0092] The embodiment of the present application makes each convolution layer correspond one-to-one with each coupling layer, thereby normalizing and guiding the output information of each convolution layer according to the guidance information output by each coupling layer, so that the posterior encoder can capture the backbone information in the linear spectrum corresponding to the training recording.

[0093] In an optional embodiment of the present application, the method further includes:

[0094] Generate a first guided coefficient corresponding to the current convolutional layer according to the guided information output by the coupling layer corresponding to the current convolutional layer;

[0095] Calculating a linear spectrum corresponding to the training recording, or a second guided coefficient corresponding to the output information of the previous convolutional layer;

[0096] Calculating according to the first guided coefficient and the second guided coefficient to obtain the output of the current convolutional layer;

[0097] If the current convolutional layer is the last convolutional layer in the posterior encoder, the training speech mean and the training speech variance are obtained according to the output of the current convolutional layer.

[0098] In this embodiment, the normalization guidance process inside each convolutional layer can be expressed as follows.

[0099] First, the first guided coefficient corresponding to the current convolutional layer is generated according to the guided information output by the coupling layer corresponding to the current convolutional layer. At the same time, the linear spectrum corresponding to the training recording, or the second guided coefficient corresponding to the output information of the previous convolutional layer, is calculated.

[0100] Afterwards, the output of the current convolution layer is obtained by calculating according to the first guided coefficient and the second guided coefficient. If the current convolution layer is the last convolution layer in the posterior encoder, the training speech mean and the training speech variance are obtained according to the output of the current convolution layer.

[0101] In one embodiment, assuming that the guidance information output by the coupling layer corresponding to the current convolutional layer is a 100x192 matrix, and the linear spectrum corresponding to the training recording or the output information of the previous convolutional layer is a 100x192 matrix, then the normalized guidance output of the current convolutional layer is a 100x192 matrix. The calculation formula for the output of the current convolutional layer can be expressed as formula (1):

[0102]

[0103] Among them, O represents the normalized guided output of the current convolutional layer, C is the guidance information of the coupling layer output corresponding to the current convolutional layer, I is the linear spectrum corresponding to the training recording or the output information of the previous convolutional layer, t represents the time dimension of the linear spectrum corresponding to the training recording [1,100], mean(I t ) and std(I t ) are respectively t The mean and standard deviation (i.e., the second bootstrap coefficient) are calculated on the 192-dimensional dimension of . W and B are two ordinary fully connected layer neural networks. W and B are used to calculate the mean and standard deviation (i.e., the second bootstrap coefficient) on the 192-dimensional dimension of . t Generate 192-dimensional affine operation coefficients and biases (ie, the first guided coefficients) on the 192-dimensional dimension of .

[0104] In one embodiment, if the linear spectrum size corresponding to the training recording is not a 100x192 matrix, it can be adjusted to a 100x192 matrix and then input into the posterior encoder.

[0105] The correlation matrix sizes and corresponding dimensions in the above embodiments are for illustrative purposes only. This application does not impose any restrictions on the setting of the relevant data, and those skilled in the art may set them according to actual needs.

[0106] Reference Figure 3 , is a normalized guidance schematic diagram of an embodiment of a speech synthesis method of the present application.

[0107] like Figure 3 As shown in FIG. 1 , the bypass guided module is composed of several coupled layers connected in series, and the posterior encoder is composed of several convolutional layers connected in series. The number of convolutional layers is the same as the number of coupled layers in the bypass guided module, and each convolutional layer corresponds to each coupling layer one by one. The number of coupled layers connected in series in the internal structure of the bypass guided module is also N LAYERS, and the number of convolutional layers connected in series in the internal structure of the posterior encoder is NLAYERS.

[0108] The training recording is input into the HuBERT model to obtain the output result, which is used as the pronunciation information corresponding to the training recording, and the pronunciation information corresponding to the training recording is used as the input of the bypass guidance module. In the bypass guidance module, when the pronunciation information passes through coupling layer 1, the guidance information of convolution layer 1 is obtained, and the output of coupling layer 1 is dimensionally flipped; the output of coupling layer 1 is dimensionally flipped and enters coupling layer 2, and the guidance information of convolution layer 2 is obtained; and so on, until the output of coupling layer NLAYERS is dimensionally flipped, and the self-supervisory loss is obtained.

[0109] In the posterior encoder, the linear spectrum is used as the main input, and the guide information is used as the bypass input. After NLAYERS convolution layers, the training speech mean μ is obtained. q and the training speech variance σ q At this time, since the guidance information is used as a bypass input and normalized and guided inside the convolutional layer, the posterior encoder can better capture the backbone information in the linear spectrum corresponding to the training recording during the data transmission process inside the convolutional layer.

[0110] The embodiment of the present application guides the posterior encoder layer by layer through a bypass guidance module, and each layer of guidance adopts a normalized guidance method, which is gradual and from shallow to deep, which is beneficial to the training stability of the overall speech synthesis model.

[0111] In an optional embodiment of the present application, the speech synthesis model further includes a decoder, a stream module and a text processing module, and the training of the speech synthesis model according to the training speech mean, the training speech variance, the self-supervisory loss and the training text includes:

[0112] Randomly sampling the Gaussian distribution model corresponding to the training speech mean and the training speech variance to obtain a speech sampling result;

[0113] Inputting the speech sampling result into the decoder to obtain the speech generation loss;

[0114] Inputting the speech sampling result into the stream module to obtain a training intermediate vector;

[0115] Inputting the training intermediate vector and the training text into the text processing module to obtain the duration loss and relative entropy;

[0116] The neural network parameters of the speech synthesis model are updated according to the self-supervision loss, the speech generation loss, the duration loss and the relative entropy.

[0117] In this embodiment, the speech synthesis model further includes a decoder, a stream module and a text processing module. The process of training the speech synthesis model according to the training speech mean, the training speech variance, the self-supervision loss and the training text can be expressed as follows.

[0118] The Gaussian distribution model corresponding to the training speech mean and the training speech variance is randomly sampled to obtain the speech sampling result. In one example, z=μ can be calculated by taking any random number rand. q +rand×σ q , z is the random sampling result of Gaussian distribution model.

[0119] The speech sampling result z is processed in two ways. One way is input into the decoder to obtain the speech generation loss; the other way is input into the stream module to obtain the training intermediate vector f θ (z).

[0120] Further training the intermediate vector f θ (z) and the training text are input into the text processing module, and the duration loss and relative entropy (ie, KL distance) are obtained through the internal processing of the text processing module.

[0121] Summarize the loss function values, add the self-supervision loss, speech generation loss, duration loss, and relative entropy as the loss function value of the entire model training, and update the neural network parameters of the speech synthesis model according to the standard neural network gradient backpropagation algorithm.

[0122] The embodiment of the present application uses the output of the posterior encoder guided by the bypass guidance module, combined with other parameters to further train the speech synthesis model, which can improve the overall training effect of the speech synthesis model.

[0123] In an optional embodiment of the present application, the text processing module includes a text encoder, a projection layer, and a random duration predictor, and the inputting of the training intermediate vector and the training text into the text processing module to obtain the duration loss and the relative entropy includes:

[0124] Inputting the training text into the text encoder to obtain a training text matrix;

[0125] Input the training text matrix into the projection layer to obtain the training text mean and the training text variance;

[0126] For the training text mean and the training text variance, the training intermediate vector is used to perform time series alignment to obtain a first training alignment parameter;

[0127] Inputting the training text matrix into the random duration predictor to obtain a second training alignment parameter;

[0128] Comparing the first training alignment parameter with the second training alignment parameter to obtain the duration loss;

[0129] The relative entropy is obtained by comparing the training text Gaussian distribution shaped by the training text mean and the training text variance with the training speech Gaussian distribution shaped by the training speech mean and the training speech variance.

[0130] In this embodiment, the text processing module of the speech synthesis model may specifically include a text encoder, a projection layer, and a random duration predictor. During the model training phase, the processing process inside the text processing module may be represented as follows.

[0131] Input the phoneme sequence corresponding to the training text into the text encoder to obtain the training text matrix h text In one example, a third-party phonetic notation tool (such as pypinyin) may be used to translate the training text into a corresponding phoneme sequence.

[0132] The training text matrix h text Input the projection layer to get the training text mean μ θ and the training text variance σ θ , training text mean μ θ and the training text variance σ θ The corresponding Gaussian distribution model can be shaped.

[0133] For the training text mean μ θ and the training text variance σ θ , using the training intermediate vector f θ (z) Perform time alignment to obtain the first training alignment parameter. In one embodiment, the MAS algorithm (a dynamic search algorithm) can be used for time alignment. For example, if the training text mean μ θ and the training text variance σ θ The length is 4, training intermediate vector f θ (z) The length is 100, and the MAS algorithm is used to correspond each moment of the latter to the target moment of the former. For example, the alignment result is 100 in length, and the alignment result is [1,1,1,1,1,1,2,2,2,3,3,3,3…4,4,4,4,4,4]; for this alignment result, the number of repetitions of the text corresponding to the speech at each moment is counted to obtain the first training alignment parameter. For example, the first moment of the text is repeated 6 times and the second moment is repeated 3 times. By repeating different moments of the text, the time series alignment of the training intermediate vector with the mean of the training text and the variance of the training text is achieved.

[0134] The training text matrix is ​​input into the random duration predictor to obtain the second training alignment parameter. The training text matrix is ​​input into the random duration predictor to obtain the second training alignment parameter, which describes the number of repetitions of the text at each moment.

[0135] The first training alignment parameter is compared with the second training alignment parameter to obtain the duration loss.

[0136] The relative entropy is obtained by comparing the training text Gaussian distribution shaped by the training text mean and training text variance with the training speech Gaussian distribution shaped by the training speech mean and training speech variance. The specific formula of relative entropy can be expressed as formula (2):

[0137]

[0138] Among them, KL distance is the relative entropy, μ θ is the mean of the training text, f θ (z) is the training intermediate vector, σ θ is the training text variance, σ q is the training speech variance.

[0139] Reference Figure 4 , is a schematic diagram of the speech synthesis model structure of an embodiment of the speech synthesis method of the present application.

[0140] Figure 4 The specific structure and training process of the speech synthesis model during the training phase are demonstrated.

[0141] Figure 3 The structure of the speech synthesis model and the flow steps of its training process of the embodiment of the present application are shown. The speech synthesis model includes a posterior encoder, a bypass guidance module, a decoder, a text encoder, a projection layer, a stream module, a timing alignment module and a random duration predictor. Among them, the posterior encoder and the bypass guidance module are only required in the training stage of the speech synthesis model, and the posterior encoder and the bypass guidance module are not needed when actually performing speech synthesis.

[0142] The phoneme sequence corresponding to the training text is first input into the text encoder, and the training text matrix h is output. text . The size of the training text matrix is ​​4xD, where D is the preset internal dimension, such as 192.

[0143] Training text matrix h text Further input the projection layer to get the training text mean μ θ and the training text variance σ θ , with sizes of 4xD respectively, which essentially shape the Gaussian distribution of 4 training texts.

[0144] On the other hand, the training recording is input into the HuBERT model to obtain the output result, which is used as the pronunciation information corresponding to the training recording, and the pronunciation information corresponding to the training recording is used as the input of the bypass guidance module, and the bypass guidance module outputs the guidance information and the self-supervision loss.

[0145] Use a third-party signal processing tool (such as torchaudio included with pyTorch) to extract the linear spectrum of the training recording. For example, if the training recording is 2 seconds long and the linear spectrum is extracted according to the settings of "frame shift 20 milliseconds, FFT window length 1024", the size of the obtained linear spectrum is 100x513.

[0146] The linear spectrum corresponding to the training recording is used as the main input, and the guidance information is used as the bypass input. Both paths are sent to the posterior encoder to obtain the training speech mean and training speech variance, both of which have a size of 100xD.

[0147] The training speech mean and training speech variance can shape the corresponding training speech Gaussian distribution, and randomly sample the training speech Gaussian distribution to obtain the speech sampling result z. The speech sampling result z needs to be output in two ways. The first way is input to the decoder to obtain the training speech waveform, and the error is calculated with the original real training recording to obtain the generation loss; the other way is output to the stream module, and after processing, the training intermediate vector f is obtained. θ (z), size is 100xD.

[0148] For training intermediate vector f θ (z) Perform temporal alignment with the training text Gaussian distribution to obtain the first training alignment parameter. Input the training text matrix into the random duration predictor to obtain the second training alignment parameter with a size of 4x1. Compare the first training alignment parameter and the second training alignment parameter to obtain the duration loss.

[0149] The relative entropy is obtained by comparing the training text Gaussian distribution shaped by the training text mean and training text variance with the training speech Gaussian distribution shaped by the training speech mean and training speech variance.

[0150] Finally, the speech synthesis model is updated using the loss function value obtained by summarizing the self-supervised loss, generation loss, duration loss, and relative entropy to obtain an updated speech synthesis model.

[0151] The embodiment of the present application configures a bypass guidance module for the posterior encoder, thereby utilizing the guidance information output by the bypass guidance module to specifically guide the posterior encoder to capture the backbone information in the linear spectrum corresponding to the training recording, which helps to improve the overall training effect of the speech synthesis model.

[0152] In an optional embodiment of the present application, inputting the target phoneme sequence corresponding to the target text into a speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model includes:

[0153] Inputting a target phoneme sequence corresponding to the target text into the text encoder to obtain a text matrix corresponding to the target phoneme sequence;

[0154] Input the text matrix into the projection layer to obtain the text mean and text variance;

[0155] Randomly sampling the text Gaussian distribution corresponding to the text mean and the text variance to obtain a text sampling result;

[0156] Inputting the text matrix into the random duration predictor to obtain alignment parameters;

[0157] According to the alignment parameters, the text sampling result is subjected to a time-to-time copy operation to obtain a target text sampling result;

[0158] Inputting the target text sampling result into the stream module to obtain an intermediate vector;

[0159] The intermediate vector is input into the decoder, and the target speech corresponding to the target text is output.

[0160] In this embodiment, the specific process of performing language synthesis using a trained speech synthesis model is introduced.

[0161] The target phoneme sequence corresponding to the target text is input into the text encoder to obtain the text matrix corresponding to the target phoneme sequence, and then the text matrix is ​​input into the projection layer to obtain the text mean and text variance.

[0162] The text Gaussian distribution corresponding to the text mean and text variance is randomly sampled to obtain the text sampling result.

[0163] The text matrix is ​​input into the random duration predictor to obtain the alignment parameter. According to the alignment parameter, the text sampling result is copied at the same time to obtain the target text sampling result, and the target text sampling result is input into the stream module to obtain the intermediate vector.

[0164] The intermediate vector is input into the decoder, and the target speech corresponding to the target text is output to complete the speech synthesis.

[0165] The speech synthesis model trained in the embodiment of the present application effectively solves the problem of inaccurate pronunciation and fluctuating sound quality of the target speech synthesized by the model, and the clarity and naturalness of the target speech are enhanced, thereby significantly improving the user experience.

[0166] In an embodiment of the present application, a target text of speech to be synthesized is received; a target phoneme sequence corresponding to the target text is input into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model. The embodiment of the present application configures a bypass guidance module for the posteriori encoder, thereby realizing the use of the guidance information output by the bypass guidance module to specifically guide the posteriori encoder to capture the backbone information in the linear spectrum corresponding to the training recording. The embodiment of the present application helps to improve the overall training effect of the speech synthesis model, effectively solves the problems of inaccurate pronunciation and fluctuating sound quality of the target speech synthesized by the model, and enhances the clarity and naturalness of the target speech, thereby significantly improving the user experience.

[0167] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0168] On the basis of the above embodiments, this embodiment further provides a speech synthesis device, which is applied to electronic devices such as terminal devices and servers.

[0169] Reference Figure 5 , shows a structural block diagram of an embodiment of a speech synthesis device of the present application, which may specifically include the following modules:

[0170] A text receiving module 501 is used to receive a target text to be synthesized into speech;

[0171] The speech synthesis module 502 is used to input the target phoneme sequence corresponding to the target text into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model.

[0172] Optionally, the bypass guide module is composed of a plurality of coupling layers connected in series, and the device comprises:

[0173] A training voice input module, used for inputting pronunciation information corresponding to the training recording into the bypass guidance module;

[0174] A guidance information acquisition module, used for acquiring the guidance information output by each coupling layer when the pronunciation information passes through the coupling layers in the bypass guidance module one by one, and acquiring the self-supervision loss output by the last coupling layer;

[0175] A normalized guidance module, used for inputting the guidance information and the linear spectrum corresponding to the training recording into the posterior encoder to obtain the training speech mean and the training speech variance;

[0176] A model training module is used to train the speech synthesis model according to the training speech mean, the training speech variance, the self-supervision loss and the training text.

[0177] Optionally, the posterior encoder is composed of a plurality of convolutional layers connected in series, the number of the convolutional layers is the same as the number of coupling layers in the bypass guidance module, each of the convolutional layers corresponds to each of the coupling layers one by one, and the normalization guidance module is further used to:

[0178] If the current convolutional layer is the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the linear spectrum corresponding to the training recording are input into the current convolutional layer;

[0179] If the current convolutional layer is not the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the output information of the previous convolutional layer are input into the current convolutional layer.

[0180] Optionally, the device further comprises:

[0181] A first guided coefficient generating module, used for generating a first guided coefficient corresponding to the current convolution layer according to the guided information output by the coupling layer corresponding to the current convolution layer;

[0182] A second guided coefficient generation module, used for calculating the linear spectrum corresponding to the training recording, or the second guided coefficient corresponding to the output information of the previous convolutional layer;

[0183] A first convolutional layer output module, configured to calculate according to the first guided coefficient and the second guided coefficient to obtain an output of the current convolutional layer;

[0184] The second convolution layer output module is used to obtain the training speech mean and the training speech variance according to the output of the current convolution layer if the current convolution layer is the last convolution layer in the posterior encoder.

[0185] Optionally, the speech synthesis model further includes a decoder, a stream module and a text processing module, and the model training module includes:

[0186] A random sampling submodule, used for randomly sampling the Gaussian distribution model corresponding to the training speech mean and the training speech variance to obtain a speech sampling result;

[0187] A decoder processing submodule, used for inputting the speech sampling result into the decoder to obtain the speech generation loss;

[0188] A normalized stream processing submodule, which can be used to input the speech sampling result into the stream module to obtain a training intermediate vector;

[0189] A text processing submodule, used for inputting the training intermediate vector and the training text into the text processing module to obtain a duration loss and a relative entropy;

[0190] A parameter updating submodule is used to update the neural network parameters of the speech synthesis model according to the self-supervision loss, the speech generation loss, the duration loss and the relative entropy.

[0191] Optionally, the text processing module includes a text encoder, a projection layer, and a random duration predictor, and the text processing submodule is further used to:

[0192] Inputting the phoneme sequence corresponding to the training text into the text encoder to obtain a training text matrix;

[0193] Input the training text matrix into the projection layer to obtain the training text mean and the training text variance;

[0194] For the training text mean and the training text variance, the training intermediate vector is used to perform time series alignment to obtain a first training alignment parameter;

[0195] Inputting the training text matrix into the random duration predictor to obtain a second training alignment parameter;

[0196] Comparing the first training alignment parameter with the second training alignment parameter to obtain the duration loss;

[0197] The relative entropy is obtained by comparing the training text Gaussian distribution shaped by the training text mean and the training text variance with the training speech Gaussian distribution shaped by the training speech mean and the training speech variance.

[0198] Optionally, the speech synthesis module 502 includes:

[0199] A target text input submodule, used for inputting a target phoneme sequence corresponding to the target text into the text encoder to obtain a text matrix corresponding to the target phoneme sequence;

[0200] A projection layer processing submodule, used for inputting the text matrix into the projection layer to obtain a text mean and a text variance;

[0201] A text sampling submodule, used for randomly sampling the text Gaussian distribution corresponding to the text mean and the text variance to obtain a text sampling result;

[0202] A random duration predictor processing submodule, used for inputting the text matrix into the random duration predictor to obtain alignment parameters;

[0203] A time copy submodule, used for performing a time copy operation on the text sampling result according to the alignment parameter to obtain a target text sampling result;

[0204] An intermediate vector generation submodule, used for inputting the target text sampling result into the stream module to obtain an intermediate vector;

[0205] The target speech synthesis submodule is used to input the intermediate vector into the decoder and output the target speech corresponding to the target text.

[0206] In an embodiment of the present application, a target text of speech to be synthesized is received; a target phoneme sequence corresponding to the target text is input into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model. The embodiment of the present application configures a bypass guidance module for the posteriori encoder, thereby realizing the use of the guidance information output by the bypass guidance module to specifically guide the posteriori encoder to capture the backbone information in the linear spectrum corresponding to the training recording. The embodiment of the present application helps to improve the overall training effect of the speech synthesis model, effectively solves the problems of inaccurate pronunciation and fluctuating sound quality of the target speech synthesized by the model, and enhances the clarity and naturalness of the target speech, thereby significantly improving the user experience.

[0207] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0208] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0209] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 6 An exemplary apparatus 600 that may be used to implement various embodiments described in this application is schematically illustrated.

[0210] For one embodiment, Figure 6 An exemplary apparatus 600 is shown having one or more processors 602, a control module (chip set) 604 coupled to at least one of the processor(s) 602, a memory 606 coupled to the control module 604, a non-volatile memory (NVM) / storage device 608 coupled to the control module 604, one or more input / output devices 610 coupled to the control module 604, and a network interface 612 coupled to the control module 604.

[0211] The processor 602 may include one or more single-core or multi-core processors, and the processor 602 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 600 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0212] In some embodiments, the apparatus 600 may include one or more computer-readable media (e.g., memory 606 or NVM / storage device 608) having instructions 614 and one or more processors 602 combined with the one or more computer-readable media and configured to execute the instructions 614 to implement a module to perform the actions described in the present disclosure.

[0213] For one embodiment, control module 604 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 602 and / or any suitable device or component in communication with control module 604 .

[0214] The control module 604 may include a memory controller module to provide an interface to the memory 606. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0215] The memory 606 may be used, for example, to load and store data and / or instructions 614 for the device 600. For one embodiment, the memory 606 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 606 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0216] For one embodiment, control module 604 may include one or more input / output controllers to provide an interface to NVM / storage device 608 and input / output device(s) 610 .

[0217] For example, NVM / storage 608 may be used to store data and / or instructions 614. NVM / storage 608 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0218] NVM / storage device 608 may include storage resources that are physically part of the device on which apparatus 600 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 608 may be accessed via input / output device(s) 610 over a network.

[0219] (One or more) input / output devices 610 may provide an interface for the apparatus 600 to communicate with any other appropriate device, and the input / output device 610 may include a communication component, an audio component, a sensor component, etc. The network interface 612 may provide an interface for the apparatus 600 to communicate through one or more networks, and the apparatus 600 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0220] For one embodiment, at least one of the processor(s) 602 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 604. For one embodiment, at least one of the processor(s) 602 may be packaged together with the logic of one or more controllers of the control module 604 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 602 may be integrated on the same die with the logic of one or more controllers of the control module 604. For one embodiment, at least one of the processor(s) 602 may be integrated on the same die with the logic of one or more controllers of the control module 604 to form a system-on-chip (SoC).

[0221] In various embodiments, the device 600 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 600 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 600 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0222] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0223] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0224] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0225] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable speech synthesis terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable speech synthesis terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0226] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable speech synthesis terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0227] These computer program instructions may also be loaded onto a computer or other programmable speech synthesis terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0228] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0229] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0230] The above is a detailed introduction to a speech synthesis method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Receiving target text for speech synthesis; The target phoneme sequence corresponding to the target text is input into a speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model.

2. The method according to claim 1, characterized in that: The bypass guidance module is composed of a plurality of coupling layers connected in series. The training process of the speech synthesis model includes: Inputting pronunciation information corresponding to the training recording into the bypass guidance module; When the pronunciation information passes through the coupling layers in the bypass guidance module one by one, the guidance information output by each coupling layer is obtained, and the self-supervision loss output by the last coupling layer is obtained; Inputting the linear spectrum corresponding to the guide information and the training recording into the posterior encoder to obtain the training speech mean and the training speech variance; The speech synthesis model is trained according to the training speech mean, the training speech variance, the self-supervision loss and the training text.

3. The method according to claim 2, characterized in that The a posteriori encoder is composed of a plurality of convolutional layers connected in series, the number of the convolutional layers is the same as the number of coupling layers in the bypass guidance module, each of the convolutional layers corresponds to each of the coupling layers one by one, and the linear spectrum corresponding to the guidance information and the training recording is input into the a posteriori encoder, including: If the current convolutional layer is the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the linear spectrum corresponding to the training recording are input into the current convolutional layer; If the current convolutional layer is not the first convolutional layer in the posterior encoder, the guidance information output by the coupling layer corresponding to the current convolutional layer and the output information of the previous convolutional layer are input into the current convolutional layer.

4. The method according to claim 3, characterized in that The method further comprises: Generate a first guided coefficient corresponding to the current convolutional layer according to the guided information output by the coupling layer corresponding to the current convolutional layer; Calculating a linear spectrum corresponding to the training recording, or a second guided coefficient corresponding to the output information of the previous convolutional layer; Calculating according to the first guided coefficient and the second guided coefficient to obtain the output of the current convolutional layer; If the current convolutional layer is the last convolutional layer in the posterior encoder, the training speech mean and the training speech variance are obtained according to the output of the current convolutional layer.

5. The method according to claim 2, characterized in that: The speech synthesis model further includes a decoder, a stream module and a text processing module. The speech synthesis model is trained according to the training speech mean, the training speech variance, the self-supervisory loss and the training text, including: Randomly sampling the Gaussian distribution model corresponding to the training speech mean and the training speech variance to obtain a speech sampling result; Inputting the speech sampling result into the decoder to obtain the speech generation loss; Inputting the speech sampling result into the stream module to obtain a training intermediate vector; Inputting the training intermediate vector and the training text into the text processing module to obtain the duration loss and relative entropy; The neural network parameters of the speech synthesis model are updated according to the self-supervision loss, the speech generation loss, the duration loss and the relative entropy.

6. The method according to claim 5, characterized in that The text processing module includes a text encoder, a projection layer, and a random duration predictor, and the training intermediate vector and the training text are input into the text processing module to obtain the duration loss and the relative entropy, including: Inputting the phoneme sequence corresponding to the training text into the text encoder to obtain a training text matrix; Input the training text matrix into the projection layer to obtain the training text mean and the training text variance; For the training text mean and the training text variance, the training intermediate vector is used to perform time series alignment to obtain a first training alignment parameter; Inputting the training text matrix into the random duration predictor to obtain a second training alignment parameter; Comparing the first training alignment parameter with the second training alignment parameter to obtain the duration loss; The relative entropy is obtained by comparing the training text Gaussian distribution shaped by the training text mean and the training text variance with the training speech Gaussian distribution shaped by the training speech mean and the training speech variance.

7. The method according to claim 6, characterized in that The step of inputting a target phoneme sequence corresponding to the target text into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model comprises: Inputting a target phoneme sequence corresponding to the target text into the text encoder to obtain a text matrix corresponding to the target phoneme sequence; Input the text matrix into the projection layer to obtain the text mean and text variance; Randomly sampling the text Gaussian distribution corresponding to the text mean and the text variance to obtain a text sampling result; Inputting the text matrix into the random duration predictor to obtain alignment parameters; According to the alignment parameters, the text sampling result is subjected to a time-to-time copy operation to obtain a target text sampling result; Inputting the target text sampling result into the stream module to obtain an intermediate vector; The intermediate vector is input into the decoder, and the target speech corresponding to the target text is output.

8. A speech synthesis device, characterized in that: The device comprises: A text receiving module, used for receiving a target text to be synthesized into speech; A speech synthesis module is used to input the target phoneme sequence corresponding to the target text into a speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posteriori encoder, a bypass guidance module, a training recording and a training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posteriori encoder so that the posteriori encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the speech synthesis method as described in one or more of claims 1-7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the speech synthesis method according to one or more of claims 1-7.

Images