A speech synthesis method, apparatus, device, and storage medium

Abstract

Embodiments of the present application provide a speech synthesis method, device, equipment and storage medium, receiving a target text of a speech to be synthesized; inputting a target phoneme sequence corresponding to the target text into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained by using a posterior encoder, a bypass guide module, a training recording and a training text; the bypass guide module is used to generate guide information; the guide information is used to guide the posterior encoder, so that the posterior encoder extracts the core speech information in the linear spectrum corresponding to the training recording, and the core speech information is used to train the speech synthesis model. Embodiments of the present application help to improve the overall training effect of the speech synthesis model, effectively solve the problem of inaccurate pronunciation and quality fluctuation of the target speech synthesized by the model, and enhance the intelligibility and naturalness of the target speech.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a speech synthesis method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] Existing speech synthesis models typically employ multi-layered, cascaded conventional convolutional neural networks as posterior encoders during training. While this design can capture the statistical characteristics of speech signals to some extent, its structural limitations result in a lack of clear directionality and focus when extracting core speech information. Specifically, conventional convolutional neural networks exhibit random transformation directions when processing speech data, failing to target the core features of the speech signal effectively and potentially becoming distracted by unimportant information. This randomness and lack of directionality in feature extraction makes it difficult for the model to consistently capture crucial information from the speech signal during training, thus impacting the overall performance of the speech synthesis model.

[0003] In practical applications, this unstable feature extraction method may lead to problems such as inaccurate pronunciation and fluctuating sound quality in the synthesized speech. For example, the model may sometimes overemphasize minor features in the speech signal while ignoring key parts of pronunciation, resulting in a decrease in the clarity and naturalness of the synthesized speech and affecting the user experience. Summary of the Invention

[0004] This application provides a speech synthesis method to solve or at least partially solve the above-mentioned problems.

[0005] Accordingly, embodiments of this application also provide a speech synthesis device, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0006] To address the aforementioned problems, this application discloses a speech synthesis method, the method comprising:

[0007] Receive the target text to be synthesized speech;

[0008] The target phoneme sequence corresponding to the target text is input into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model. The speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training text. The bypass guidance module is used to generate guidance information. The guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording. The backbone speech information is used to train the speech synthesis model.

[0009] Optionally, the bypass guidance module consists of several coupling layers connected in series, and the training process of the speech synthesis model includes:

[0010] The pronunciation information corresponding to the training recording is input into the bypass guidance module;

[0011] As the pronunciation information passes through the coupling layers in the bypass guidance module one by one, the guidance information output by each coupling layer is obtained, and the self-supervised loss output by the last coupling layer is obtained;

[0012] The guidance information and the linear spectrum corresponding to the training recording are input into the posterior encoder to obtain the mean and variance of the training speech.

[0013] The speech synthesis model is trained based on the mean of the training speech, the variance of the training speech, the self-supervised loss, and the training text.

[0014] Optionally, the posterior encoder consists of several convolutional layers connected in series, the number of which is the same as the number of coupling layers in the bypass guidance module, and each convolutional layer corresponds one-to-one with each coupling layer. The step of inputting the guidance information and the linear spectrum corresponding to the training recording into the posterior encoder includes:

[0015] If the current convolutional layer is the first convolutional layer in the posterior encoder, then the guidance information output by the coupling layer corresponding to the current convolutional layer, and the linear spectrum corresponding to the training recording, are input into the current convolutional layer.

[0016] If the current convolutional layer is not the first convolutional layer in the posterior encoder, then the guiding information output by the coupling layer corresponding to the current convolutional layer, as well as the output information of the previous convolutional layer, are input into the current convolutional layer.

[0017] Optionally, the method further includes:

[0018] Based on the guiding information output by the coupling layer corresponding to the current convolutional layer, generate the first guiding coefficient corresponding to the current convolutional layer;

[0019] Calculate the linear spectrum corresponding to the training recording, or the second guiding coefficient corresponding to the output information of the previous convolutional layer;

[0020] The output of the current convolutional layer is calculated based on the first guiding coefficient and the second guiding coefficient.

[0021] If the current convolutional layer is the last convolutional layer in the posterior encoder, then the mean and variance of the training speech are obtained based on the output of the current convolutional layer.

[0022] Optionally, the speech synthesis model further includes a decoder, a streaming module, and a text processing module. Training the speech synthesis model based on the training speech mean, the training speech variance, the self-supervised loss, and the training text includes:

[0023] Random sampling is performed on the Gaussian distribution model corresponding to the mean and variance of the training speech to obtain the speech sampling results;

[0024] The speech sampling results are input into the decoder to obtain the speech generation loss;

[0025] The speech sampling results are input into the streaming module to obtain the training intermediate vector;

[0026] The training intermediate vector and the training text are input into the text processing module to obtain the duration loss and relative entropy.

[0027] The neural network parameters of the speech synthesis model are updated based on the self-supervised loss, the speech generation loss, the duration loss, and the relative entropy.

[0028] Optionally, the text processing module includes a text encoder, a projection layer, and a random duration predictor. The step of inputting the training intermediate vector and the training text into the text processing module to obtain the duration loss and relative entropy includes:

[0029] The phoneme sequence corresponding to the training text is input into the text encoder to obtain the training text matrix;

[0030] The training text matrix is ​​input into the projection layer to obtain the mean and variance of the training text.

[0031] For the mean and variance of the training text, the training intermediate vector is used for temporal alignment to obtain the first training alignment parameter.

[0032] The training text matrix is ​​input into the random duration predictor to obtain the second training alignment parameters;

[0033] The first training alignment parameter and the second training alignment parameter are compared to obtain the duration loss;

[0034] The relative entropy is obtained by comparing the Gaussian distribution of the training text shaped by the mean and variance of the training text with the Gaussian distribution of the training speech shaped by the mean and variance of the training speech.

[0035] Optionally, the step of inputting the target phoneme sequence corresponding to the target text into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model includes:

[0036] The target phoneme sequence corresponding to the target text is input into the text encoder to obtain the text matrix corresponding to the target phoneme sequence;

[0037] The text matrix is ​​input into the projection layer to obtain the text mean and text variance.

[0038] Randomly sample the Gaussian distribution of the text mean and the text variance to obtain the text sampling results;

[0039] The text matrix is ​​input into the random duration predictor to obtain the alignment parameters;

[0040] Based on the alignment parameters, the text sampling result is copied at any time to obtain the target text sampling result;

[0041] The target text sampling result is input into the streaming module to obtain an intermediate vector;

[0042] The intermediate vector is input into the decoder, which outputs the target speech corresponding to the target text.

[0043] This application also discloses a speech synthesis device, the device comprising:

[0044] The text receiving module is used to receive the target text of the speech to be synthesized.

[0045] A speech synthesis module is used to input the target phoneme sequence corresponding to the target text into a speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model. The speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training text. The bypass guidance module is used to generate guidance information. The guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording. The backbone speech information is used to train the speech synthesis model.

[0046] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more speech synthesis methods as described in the embodiments of this application.

[0047] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform one or more of the speech synthesis methods described in this application.

[0048] Compared with the prior art, the embodiments of this application have the following advantages:

[0049] In this embodiment, the target text to be synthesized is received; the target phoneme sequence corresponding to the target text is input into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model. This embodiment, by configuring a bypass guidance module for the posterior encoder, realizes the use of guidance information output by the bypass guidance module to specifically guide the posterior encoder to capture the backbone information in the linear spectrum corresponding to the training recording. This embodiment helps improve the overall training effect of the speech synthesis model, effectively solves the problems of inaccurate pronunciation and fluctuating sound quality of the synthesized target speech, enhances the clarity and naturalness of the target speech, and thus significantly improves the user experience. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the steps of an embodiment of a speech synthesis method according to this application;

[0051] Figure 2 This is a schematic diagram of the coupling layer of an embodiment of the speech synthesis method of this application;

[0052] Figure 3 This is a normalized guidance diagram of an embodiment of the speech synthesis method of this application;

[0053] Figure 4 This is a schematic diagram of the speech synthesis model structure of an embodiment of the speech synthesis method of this application;

[0054] Figure 5 This is a structural block diagram of an embodiment of a speech synthesis device according to this application;

[0055] Figure 6 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0056] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Reference Figure 1 is a flowchart of the steps of an embodiment of a speech synthesis method according to this application, including the following steps:

[0058] Step 101: Receive the target text of the speech to be synthesized.

[0059] In step 101, before using the speech synthesis model to synthesize the target speech, it is necessary to first receive the target text of the speech to be synthesized.

[0060] In one embodiment, a third-party phonetic annotation tool (such as pypinyin, a Python library for converting Chinese characters into pinyin) can be used first to translate the target text of the speech to be synthesized into a target phoneme sequence. In one example, the target phoneme sequence for Chinese is the initials and finals, for example, the target phoneme sequence of "good person" is "h3ao3r2en2".

[0061] Step 102: Input the target phoneme sequence corresponding to the target text into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training texts; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recordings, and the backbone speech information is used to train the speech synthesis model.

[0062] In step 102, the target phoneme sequence translated from the target text of the speech to be synthesized is input into the speech synthesis model, so as to obtain the target speech corresponding to the target text output by the speech synthesis model.

[0063] In this embodiment, before using the speech synthesis model, it is necessary to first train the speech synthesis model. Specifically, when training the speech synthesis model, a posterior encoder, a bypass guidance module, training recordings, and training texts are required for training.

[0064] Existing posterior encoders usually adopt a conventional convolutional neural network structure with multiple layers in series. This structure theoretically forms a statistical transformation, but its direction is random and lacks clear directivity. It does not specifically extract the backbone information in the speech and may deviate to unimportant information. This uncertainty leads to unstable performance of the overall speech synthesis model trained and problems such as inaccurate pronunciation and fluctuating sound quality in the synthesized speech.

[0065] This embodiment addresses this issue by introducing a bypass guidance module. The bypass guidance module generates guidance information, which guides the posterior encoder to extract backbone speech information from the linear spectrum corresponding to the training recording. This backbone speech information can then be used to train the speech synthesis model. For example, using guidance information to guide the posterior encoder allows it to better capture the backbone speech information most crucial for speech synthesis, such as phoneme information, pitch information, prosody information, and spectral features, thus resulting in a more stable training effect for the speech synthesis model.

[0066] In one embodiment, the training recording can be input into the HuBERT model (Hidden-Unit BERT, a speech representation model based on self-supervised learning) to obtain the output. This output is then used as the pronunciation information corresponding to the training recording, which is then used as the input to a bypass guidance module, enabling the bypass guidance module to output guidance information. Since the output of the HuBERT model inherently reflects the core information in the training speech, using it as the input to the bypass guidance module allows the module's output to assist the posterior encoder in better capturing the most crucial backbone speech information for speech synthesis within the linear spectrum.

[0067] In this embodiment, the target text to be synthesized is received; the target phoneme sequence corresponding to the target text is input into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model. This embodiment, by configuring a bypass guidance module for the posterior encoder, realizes the use of guidance information output by the bypass guidance module to specifically guide the posterior encoder to capture the backbone information in the linear spectrum corresponding to the training recording. This embodiment helps improve the overall training effect of the speech synthesis model, effectively solves the problems of inaccurate pronunciation and fluctuating sound quality of the synthesized target speech, enhances the clarity and naturalness of the target speech, and thus significantly improves the user experience.

[0068] In one optional embodiment of this application, the bypass guidance module consists of several coupling layers connected in series, and the training process of the speech synthesis model includes:

[0069] The pronunciation information corresponding to the training recording is input into the bypass guidance module;

[0070] As the pronunciation information passes through the coupling layers in the bypass guidance module one by one, the guidance information output by each coupling layer is obtained, and the self-supervised loss output by the last coupling layer is obtained;

[0071] The guidance information and the linear spectrum corresponding to the training recording are input into the posterior encoder to obtain the mean and variance of the training speech.

[0072] The speech synthesis model is trained based on the mean of the training speech, the variance of the training speech, the self-supervised loss, and the training text.

[0073] This embodiment further illustrates the process of training a speech synthesis model using a bypass guidance module. The internal structure of the bypass guidance module in this embodiment adopts a "normalized flow" approach, meaning the bypass guidance module consists of several coupling layers connected in series.

[0074] Reference Figure 2 This is a schematic diagram of the coupling layer of an embodiment of a speech synthesis method of this application.

[0075] Figure 2 The diagram illustrates the specific operation of a coupling layer within the bypass bootstrapping module. This coupling layer structure enables both forward transformations from x to z and inverse transformations from z to x. Figure 2 The diagram shows the process of the forward transformation from x to z used during the training of the speech synthesis model.

[0076] Specifically, if both signals x and z have a dimension of D, then by pre-setting a constant d (d < D), the D-dimensional signal can be divided into two parts: the upper part consisting of x1, ..., xz. d and the lower half of x d+1 ... x D In the coupling layer transformation, the upper part remains unchanged and can be directly derived from x1, ..., x d Copying yields z1, ..., z d .

[0077] The lower part performs an affine transformation, with the coefficients and biases calculated from the upper part. For example... Figure 2 As shown, β d+1 ..., β D The coefficients of the affine transformation are calculated using the neural network submodule F; γ d+1 ..., γ D The bias for the affine transformation is calculated using the neural network submodule H. d+1 ... x D z can be obtained after affine transformation. d+1 ... zD .

[0078] Therefore, with each coupling layer transformation, only a portion of the dimensions changes, while the remaining dimensions remain unchanged. Introducing a dimension flipping operation after each coupling layer transformation swaps the changed and unchanged parts before proceeding to the next coupling layer. In this way, by serially transforming an even number of coupling layers, both parts of the dimensions undergo approximately the same number of transformations.

[0079] In this embodiment, the training process of the speech synthesis model can be specifically represented as follows.

[0080] First, the pronunciation information corresponding to the training recording is input into the bypass guidance module. In one embodiment, the training recording can be input into the HuBERT model to obtain the output result, and this output result can be used as the pronunciation information corresponding to the training recording, thereby using the pronunciation information corresponding to the training recording as the input to the bypass guidance module.

[0081] Next, as the pronunciation information passes through the coupling layers in the bypass guidance module one by one, the guidance information output by each coupling layer is obtained, as well as the self-supervised loss output by the last coupling layer. Specifically, the outputs of each coupling layer can be extracted in the middle of the bypass guidance module, and these outputs extracted from each coupling layer are finally aggregated to form a sequence of length N LAYERS, which serves as the guidance information. Here, N LAYERS represents the number of coupling layers connected in series within the internal structure of the bypass guidance module. Finally, the self-supervised loss can be output at the end of the bypass guidance module.

[0082] In one embodiment, the posterior encoder consists of several convolutional layers connected in series, with the number of convolutional layers being the same as the number of coupling layers in the bypass guidance module, and each convolutional layer corresponding to each coupling layer. Therefore, the output of each coupling layer in the bypass guidance module is actually the guidance information required by each convolutional layer in the posterior encoder.

[0083] Next, the linear spectrum corresponding to the guidance information and the training recording is input into the posterior encoder to obtain the training speech mean and training speech variance. In one embodiment, the linear spectrum can be used as the main input, while the guidance information is used as a bypass input, and both are fed into the posterior encoder to obtain the training speech mean and training speech variance. These training speech means and training speech variances can shape the corresponding Gaussian distribution model.

[0084] The speech synthesis model is trained using the training speech mean, training speech variance, self-supervised loss, and training text.

[0085] This application embodiment configures a bypass guidance module for the posterior encoder. The guidance information output by the bypass guidance module can be used to specifically guide the posterior encoder to capture the backbone information in the linear spectrum corresponding to the training recording.

[0086] In an optional embodiment of this application, the posterior encoder consists of a plurality of convolutional layers connected in series, the number of convolutional layers being the same as the number of coupling layers in the bypass guidance module, and each convolutional layer corresponding one-to-one with each coupling layer. The step of inputting the guidance information and the linear spectrum corresponding to the training recording into the posterior encoder includes:

[0087] If the current convolutional layer is the first convolutional layer in the posterior encoder, then the guidance information output by the coupling layer corresponding to the current convolutional layer, and the linear spectrum corresponding to the training recording, are input into the current convolutional layer.

[0088] If the current convolutional layer is not the first convolutional layer in the posterior encoder, then the guiding information output by the coupling layer corresponding to the current convolutional layer, as well as the output information of the previous convolutional layer, are input into the current convolutional layer.

[0089] In this embodiment, the posterior encoder consists of several convolutional layers connected in series. The number of convolutional layers is the same as the number of coupling layers in the bypass guidance module. Each convolutional layer corresponds one-to-one with each coupling layer; that is, the output of each coupling layer in the bypass guidance module is actually the guidance information required by each convolutional layer in the posterior encoder. In one example, the number of coupling layers connected in series in the internal structure of the bypass guidance module is also N LAYERS, then the number of convolutional layers connected in series in the internal structure of the posterior encoder is NLAYERS.

[0090] Specifically, the guidance information output by the coupling layer can be used to normalize and guide the output of the convolutional layer within the convolutional layer itself. If the current convolutional layer is the first convolutional layer in the posterior encoder structure, the guidance information output by the coupling layer corresponding to the current convolutional layer, along with the linear spectrum corresponding to the training recording, is input into the current convolutional layer. In one example, if the current convolutional layer is the first convolutional layer in the posterior encoder structure, then the coupling layer corresponding to the current convolutional layer is also the first coupling layer in the bypass guidance module. The guidance information output by the first coupling layer in the bypass guidance module is obtained from the guidance information, and this guidance information, along with the linear spectrum corresponding to the training recording, is input into the current convolutional layer.

[0091] If the current convolutional layer is not the first convolutional layer in the posterior encoder, then the guidance information output by the coupling layer corresponding to the current convolutional layer, along with the output information of the previous convolutional layer, is input into the current convolutional layer. In one example, if the current convolutional layer is the second convolutional layer in the posterior encoder structure, then the coupling layer corresponding to the current convolutional layer is also the second coupling layer in the bypass guidance module. The guidance information output by the second coupling layer in the bypass guidance module is obtained from the guidance information, and this guidance information, along with the output information of the first convolutional layer, is input into the second convolutional layer.

[0092] This application embodiment establishes a one-to-one correspondence between each convolutional layer and each coupling layer, thereby normalizing and guiding the output information of each convolutional layer based on the guidance information output by each coupling layer, so that the posterior encoder can capture the backbone information in the linear spectrum corresponding to the training recording.

[0093] In an optional embodiment of this application, the method further includes:

[0094] Based on the guiding information output by the coupling layer corresponding to the current convolutional layer, generate the first guiding coefficient corresponding to the current convolutional layer;

[0095] Calculate the linear spectrum corresponding to the training recording, or the second guiding coefficient corresponding to the output information of the previous convolutional layer;

[0096] The output of the current convolutional layer is calculated based on the first guiding coefficient and the second guiding coefficient.

[0097] If the current convolutional layer is the last convolutional layer in the posterior encoder, then the mean and variance of the training speech are obtained based on the output of the current convolutional layer.

[0098] In this embodiment, the normalization guidance process within each convolutional layer can be represented as follows.

[0099] First, based on the guiding information output by the coupling layer corresponding to the current convolutional layer, the first guiding coefficient for the current convolutional layer is generated. Simultaneously, the linear spectrum corresponding to the training recording, or the second guiding coefficient corresponding to the output information of the previous convolutional layer, is calculated.

[0100] Then, the output of the current convolutional layer is calculated based on the first and second guiding coefficients. If the current convolutional layer is the last convolutional layer in the posterior encoder, the mean and variance of the training speech are obtained based on the output of the current convolutional layer.

[0101] In one embodiment, assuming the guiding information output by the coupling layer corresponding to the current convolutional layer is a 100x192 matrix, and the linear spectrum corresponding to the training recording or the output information of the previous convolutional layer is a 100x192 matrix, then the normalized guiding output of the current convolutional layer is a 100x192 matrix. The calculation formula for the output of the current convolutional layer can be expressed as formula (1):

[0102]

[0103] Where O represents the normalized guided output of the current convolutional layer, C is the guided information output of the coupling layer corresponding to the current convolutional layer, I is the linear spectrum corresponding to the training recording or the output information of the previous convolutional layer, t represents the time dimension of the linear spectrum corresponding to the training recording [1, 100], and mean(I t ) and std(I t ) are respectively for I t The mean and standard deviation (i.e., the second guiding coefficients) are calculated over a 192-dimensional network. W and B are two ordinary fully connected layers in a neural network. W and B are used to calculate the mean and standard deviation over a 192-dimensional network. t Generate 192-dimensional affine operation coefficients and biases (i.e., first guiding coefficients) on the 192-dimensional dimension.

[0104] In one embodiment, if the linear spectrum size corresponding to the training recording is not a 100x192 matrix, it can be adjusted to a 100x192 matrix before being input into the posterior encoder.

[0105] The matrix dimensions and corresponding dimensions in the above embodiments are merely examples. This application does not impose any restrictions on the setting of related data, and those skilled in the art can set them according to actual needs.

[0106] Reference Figure 3 This is a normalized guidance diagram of an embodiment of a speech synthesis method of this application.

[0107] like Figure 3 As shown, the bypass guidance module consists of several coupling layers connected in series, and the posterior encoder consists of several convolutional layers connected in series. The number of convolutional layers is the same as the number of coupling layers in the bypass guidance module. Each convolutional layer corresponds one-to-one with each coupling layer. The number of coupling layers connected in series in the internal structure of the bypass guidance module is also N LAYERS. Therefore, the number of convolutional layers connected in series in the internal structure of the posterior encoder is NLAYERS.

[0108] The training recordings are input into the HuBERT model to obtain the output, which is then used as the pronunciation information corresponding to the training recordings. This pronunciation information is then used as the input to the bypass guidance module. In the bypass guidance module, when the pronunciation information passes through coupling layer 1, it receives guidance information from convolutional layer 1, and the output of coupling layer 1 undergoes dimension flipping. After dimension flipping, the output of coupling layer 1 enters coupling layer 2, where it receives guidance information from convolutional layer 2. This process continues until the output of coupling layer NLAYERS undergoes dimension flipping, resulting in the self-supervised loss.

[0109] In the posterior encoder, the linear spectrum is used as the main input, while the guiding information is used as a bypass input. After passing through NLAYERS convolutional layers, the mean μ of the training speech is obtained. q With training speech variance σ q At this point, since the guiding information is used as a bypass input and is normalized and guided inside the convolutional layer, the posterior encoder can better capture the backbone information in the linear spectrum corresponding to the training recording during the data transmission process inside the convolutional layer.

[0110] In this embodiment, the posterior encoder is guided layer by layer through a bypass guidance module. Each layer is guided using a normalized guidance method, proceeding gradually from shallow to deep, which is beneficial to the training stability of the overall speech synthesis model.

[0111] In an optional embodiment of this application, the speech synthesis model further includes a decoder, a streaming module, and a text processing module. Training the speech synthesis model based on the training speech mean, the training speech variance, the self-supervised loss, and the training text includes:

[0112] Random sampling is performed on the Gaussian distribution model corresponding to the mean and variance of the training speech to obtain the speech sampling results;

[0113] The speech sampling results are input into the decoder to obtain the speech generation loss;

[0114] The speech sampling results are input into the streaming module to obtain the training intermediate vector;

[0115] The training intermediate vector and the training text are input into the text processing module to obtain the duration loss and relative entropy.

[0116] The neural network parameters of the speech synthesis model are updated based on the self-supervised loss, the speech generation loss, the duration loss, and the relative entropy.

[0117] In this embodiment, the speech synthesis model further includes a decoder, a streaming module, and a text processing module. The process of training the speech synthesis model based on the training speech mean, training speech variance, self-supervised loss, and training text can be represented as follows.

[0118] The Gaussian distribution model corresponding to the training speech mean and variance is randomly sampled to obtain the speech sampling results. In one example, z = μ can be calculated by arbitrarily selecting a random number rand. q +rand×σ q z represents the random sampling result of the Gaussian distribution model.

[0119] The speech sampling result z is processed in two paths: one path is input into the decoder to obtain the speech generation loss; the other path is input into the streaming module to obtain the training intermediate vector f. θ (z).

[0120] Further training the intermediate vector f θ (z) and the training text input text processing module, through the internal processing of the text processing module, obtain the duration loss and relative entropy (i.e., KL distance).

[0121] The loss function values ​​are summed by adding the self-supervised loss, speech generation loss, duration loss, and relative entropy, and used as the loss function value for the entire model training. The neural network parameters of the speech synthesis model are then updated according to the standard neural network gradient backpropagation algorithm.

[0122] The embodiments of this application use the output of the posterior encoder guided by the bypass guidance module, combined with other parameters to further train the speech synthesis model, which can improve the overall training effect of the speech synthesis model.

[0123] In an optional embodiment of this application, the text processing module includes a text encoder, a projection layer, and a random duration predictor. The step of inputting the training intermediate vector and the training text into the text processing module to obtain the duration loss and relative entropy includes:

[0124] The training text is input into the text encoder to obtain the training text matrix;

[0125] The training text matrix is ​​input into the projection layer to obtain the mean and variance of the training text.

[0126] For the mean and variance of the training text, the training intermediate vector is used for temporal alignment to obtain the first training alignment parameter.

[0127] The training text matrix is ​​input into the random duration predictor to obtain the second training alignment parameters;

[0128] The first training alignment parameter and the second training alignment parameter are compared to obtain the duration loss;

[0129] The relative entropy is obtained by comparing the Gaussian distribution of the training text shaped by the mean and variance of the training text with the Gaussian distribution of the training speech shaped by the mean and variance of the training speech.

[0130] In this embodiment, the text processing module of the speech synthesis model may specifically include a text encoder, a projection layer, and a random duration predictor. During the model training phase, the internal processing of the text processing module can be represented as follows.

[0131] Input the phoneme sequence corresponding to the training text into the text encoder to obtain the training text matrix h. text In one example, a third-party phonetic transcription tool (such as pypinyin) can be used to transcribe the training text into the corresponding phoneme sequence.

[0132] The training text matrix h text Input the projection layer to obtain the mean μ of the training text. θ The variance σ of the training text θ The mean of the training text, μ θ The variance σ of the training text θ It can create a corresponding Gaussian distribution model.

[0133] For the mean μ of the training text θ The variance σ of the training text θ Using the training intermediate vector f θ (z) Perform temporal alignment to obtain the first training alignment parameters. In one embodiment, the MAS algorithm (a dynamic search algorithm) can be used for temporal alignment. For example, if the training text mean μ θ The variance σ of the training text θ The length of each vector is 4, and the intermediate vector f is trained. θ (z) has a length of 100. The MAS algorithm is used to map each time step of the latter to the target time step in the former. For example, if the alignment result has a length of 100, the alignment result is [1,1,1,1,1,1,2,2,2,3,3,3,3…4,4,4,4,4,4]. For this alignment result, the number of times the text corresponds to the speech is counted to obtain the first training alignment parameter. For example, the first time step of the text is repeated 6 times, and the second time step is repeated 3 times. By repeating different times of the text, the training intermediate vector is temporally aligned with the training text mean and the training text variance.

[0134] The training text matrix is ​​input into the random duration predictor to obtain the second training alignment parameter, which describes the number of repetitions of the text at each time step.

[0135] The first training alignment parameter and the second training alignment parameter are compared to obtain the duration loss.

[0136] The relative entropy is obtained by comparing the Gaussian distribution of the training text shaped by the mean and variance of the training text with the Gaussian distribution of the training speech shaped by the mean and variance of the training speech. The specific formula for the relative entropy can be expressed as formula (2):

[0137]

[0138] Wherein, the KL distance is the relative entropy, μ θ To train the text mean, f θ (z) is the intermediate training vector, σ θ To train the text variance, σ q To train speech variance.

[0139] Reference Figure 4 This is a schematic diagram of the speech synthesis model structure of an embodiment of the speech synthesis method of this application.

[0140] Figure 4 It demonstrates the specific structure and training process of the speech synthesis model during the training phase.

[0141] Figure 3 The diagram illustrates the structure of a speech synthesis model and the flowchart of its training process according to an embodiment of this application. The speech synthesis model includes a posterior encoder, a bypass guidance module, a decoder, a text encoder, a projection layer, a streaming module, a temporal alignment module, and a random duration predictor. The posterior encoder and bypass guidance module are only required during the training phase of the speech synthesis model; they are not needed during actual speech synthesis.

[0142] The phoneme sequence corresponding to the training text is first input into the text encoder, and the output is the training text matrix h. text The training text matrix has a size of 4xD, where D is a preset internal dimension, such as 192.

[0143] Training text matrix h text The input is further fed into the projection layer to obtain the mean μ of the training text. θ and training text variance σ θ The dimensions are 4xD, and they essentially shape four Gaussian distributions of the training texts.

[0144] On the other hand, the training recording is input into the HuBERT model to obtain the output result. This output result is used as the pronunciation information corresponding to the training recording, and the pronunciation information corresponding to the training recording is used as the input of the bypass guidance module. The bypass guidance module outputs guidance information and self-supervised loss.

[0145] Use a third-party signal processing tool (such as torchaudio, which comes with pyTorch) to extract the linear spectrum of the training recording. For example, if the training recording is 2 seconds long and the linear spectrum is extracted with the settings "frame shift 20 milliseconds, FFT window size 1024", the resulting linear spectrum will be 100x513 pixels.

[0146] The linear spectrum corresponding to the training recording is used as the main input, while the guidance information is used as the bypass input. Both are fed into the posterior encoder to obtain the mean and variance of the training speech, both with a size of 100xD.

[0147] The mean and variance of the training speech data shape a corresponding Gaussian distribution for the training speech. Random sampling is performed on this Gaussian distribution to obtain the speech sampling result z. This z sample result needs to be output in two paths. The first path is input to the decoder to obtain the training speech waveform, which is then compared with the original, real training recording to calculate the error and obtain the generation loss. The second path is output to the streaming module, where it is processed to obtain the intermediate training vector f. θ (z), with dimensions of 100xD.

[0148] For the training intermediate vector f θ (z) Time-align the training text with the Gaussian distribution to obtain the first training alignment parameter. Input the training text matrix into the random duration predictor to obtain the second training alignment parameter, with a size of 4x1. Compare the first and second training alignment parameters to obtain the duration loss.

[0149] The relative entropy is obtained by comparing the Gaussian distribution of the training text shaped by the mean and variance of the training text with the Gaussian distribution of the training speech shaped by the mean and variance of the training speech.

[0150] Finally, the speech synthesis model is updated using the loss function value obtained by summing the self-supervised loss, generation loss, duration loss, and relative entropy, thus obtaining the updated speech synthesis model.

[0151] This application embodiment configures a bypass guidance module for the posterior encoder, enabling the use of guidance information output by the bypass guidance module to specifically guide the posterior encoder to capture backbone information in the linear spectrum corresponding to the training recording, which helps to improve the overall training effect of the speech synthesis model.

[0152] In an optional embodiment of this application, the step of inputting the target phoneme sequence corresponding to the target text into a speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model includes:

[0153] The target phoneme sequence corresponding to the target text is input into the text encoder to obtain the text matrix corresponding to the target phoneme sequence;

[0154] The text matrix is ​​input into the projection layer to obtain the text mean and text variance.

[0155] Randomly sample the Gaussian distribution of the text mean and the text variance to obtain the text sampling results;

[0156] The text matrix is ​​input into the random duration predictor to obtain the alignment parameters;

[0157] Based on the alignment parameters, the text sampling result is copied at any time to obtain the target text sampling result;

[0158] The target text sampling result is input into the streaming module to obtain an intermediate vector;

[0159] The intermediate vector is input into the decoder, which outputs the target speech corresponding to the target text.

[0160] This embodiment describes the specific process of using a trained speech synthesis model for language synthesis.

[0161] The target phoneme sequence corresponding to the target text is input into the text encoder to obtain the text matrix corresponding to the target phoneme sequence. The text matrix is ​​then input into the projection layer to obtain the text mean and text variance.

[0162] Random sampling is performed on the Gaussian distribution of the text mean and text variance to obtain the text sampling results.

[0163] The text matrix is ​​input into the random duration predictor to obtain alignment parameters. Based on the alignment parameters, a time-duration copying operation is performed on the text sampling results to obtain the target text sampling results. The target text sampling results are then input into the stream module to obtain the intermediate vector.

[0164] The intermediate vector is input into the decoder, which outputs the target speech corresponding to the target text, thus completing speech synthesis.

[0165] The trained speech synthesis model in this application effectively solves the problems of inaccurate pronunciation and fluctuating sound quality in the synthesized target speech, enhancing the clarity and naturalness of the target speech and thus significantly improving the user experience.

[0166] In this embodiment, the target text to be synthesized is received; the target phoneme sequence corresponding to the target text is input into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model. This embodiment, by configuring a bypass guidance module for the posterior encoder, realizes the use of guidance information output by the bypass guidance module to specifically guide the posterior encoder to capture the backbone information in the linear spectrum corresponding to the training recording. This embodiment helps improve the overall training effect of the speech synthesis model, effectively solves the problems of inaccurate pronunciation and fluctuating sound quality of the synthesized target speech, enhances the clarity and naturalness of the target speech, and thus significantly improves the user experience.

[0167] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0168] Based on the above embodiments, this embodiment also provides a speech synthesis device that can be applied to electronic devices such as terminal devices and servers.

[0169] Reference Figure 5 The diagram shows a structural block diagram of an embodiment of a speech synthesis device according to this application, which may specifically include the following modules:

[0170] Text receiving module 501 is used to receive the target text of the speech to be synthesized;

[0171] The speech synthesis module 502 is used to input the target phoneme sequence corresponding to the target text into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model.

[0172] Optionally, the bypass guidance module is composed of several coupling layers connected in series, and the device includes:

[0173] A training voice input module is used to input the pronunciation information corresponding to the training recording into the bypass guidance module;

[0174] The guidance information acquisition module is used to acquire the guidance information output by each coupling layer when the pronunciation information passes through the coupling layers in the bypass guidance module one by one, and to acquire the self-supervised loss output by the last coupling layer.

[0175] The normalization guidance module is used to input the guidance information and the linear spectrum corresponding to the training recording into the posterior encoder to obtain the mean and variance of the training speech.

[0176] The model training module is used to train the speech synthesis model based on the training speech mean, the training speech variance, the self-supervised loss, and the training text.

[0177] Optionally, the posterior encoder consists of several convolutional layers connected in series, the number of convolutional layers being the same as the number of coupling layers in the bypass guidance module, with each convolutional layer corresponding one-to-one with each coupling layer. The normalization guidance module is further configured to:

[0178] If the current convolutional layer is the first convolutional layer in the posterior encoder, then the guidance information output by the coupling layer corresponding to the current convolutional layer, and the linear spectrum corresponding to the training recording, are input into the current convolutional layer.

[0179] If the current convolutional layer is not the first convolutional layer in the posterior encoder, then the guiding information output by the coupling layer corresponding to the current convolutional layer, as well as the output information of the previous convolutional layer, are input into the current convolutional layer.

[0180] Optionally, the device further includes:

[0181] The first guiding coefficient generation module is used to generate the first guiding coefficient corresponding to the current convolutional layer based on the guiding information output by the coupling layer corresponding to the current convolutional layer.

[0182] The second guiding coefficient generation module is used to calculate the linear spectrum corresponding to the training recording, or the second guiding coefficient corresponding to the output information of the previous convolutional layer.

[0183] The first convolutional layer output module is used to calculate the output of the current convolutional layer based on the first guiding coefficient and the second guiding coefficient.

[0184] The second convolutional layer output module is used to obtain the mean and variance of the training speech based on the output of the current convolutional layer if the current convolutional layer is the last convolutional layer in the posterior encoder.

[0185] Optionally, the speech synthesis model further includes a decoder, a streaming module, and a text processing module, and the model training module includes:

[0186] The random sampling submodule is used to randomly sample the Gaussian distribution model corresponding to the mean and variance of the training speech to obtain the speech sampling result.

[0187] The decoder processing submodule is used to input the speech sampling results into the decoder to obtain the speech generation loss;

[0188] The normalized stream processing submodule can be used to input the speech sampling results into the stream module to obtain training intermediate vectors;

[0189] The text processing submodule is used to input the training intermediate vector and the training text into the text processing module to obtain the duration loss and relative entropy;

[0190] The parameter update submodule is used to update the neural network parameters of the speech synthesis model based on the self-supervised loss, the speech generation loss, the duration loss, and the relative entropy.

[0191] Optionally, the text processing module includes a text encoder, a projection layer, and a random duration predictor. The text processing submodule is further configured to:

[0192] The phoneme sequence corresponding to the training text is input into the text encoder to obtain the training text matrix;

[0193] The training text matrix is ​​input into the projection layer to obtain the mean and variance of the training text.

[0194] For the mean and variance of the training text, the training intermediate vector is used for temporal alignment to obtain the first training alignment parameter.

[0195] The training text matrix is ​​input into the random duration predictor to obtain the second training alignment parameters;

[0196] The first training alignment parameter and the second training alignment parameter are compared to obtain the duration loss;

[0197] The relative entropy is obtained by comparing the Gaussian distribution of the training text shaped by the mean and variance of the training text with the Gaussian distribution of the training speech shaped by the mean and variance of the training speech.

[0198] Optionally, the speech synthesis module 502 includes:

[0199] The target text input submodule is used to input the target phoneme sequence corresponding to the target text into the text encoder to obtain the text matrix corresponding to the target phoneme sequence;

[0200] The projection layer processing submodule is used to input the text matrix into the projection layer to obtain the text mean and text variance.

[0201] The text sampling submodule is used to randomly sample the text mean and the text variance corresponding to the text Gaussian distribution to obtain the text sampling result;

[0202] The random duration predictor processing submodule is used to input the text matrix into the random duration predictor to obtain alignment parameters;

[0203] The time-copy submodule is used to perform a time-copy operation on the text sampling result according to the alignment parameters to obtain the target text sampling result;

[0204] The intermediate vector generation submodule is used to input the target text sampling result into the stream module to obtain the intermediate vector;

[0205] The target speech synthesis submodule is used to input the intermediate vector into the decoder and output the target speech corresponding to the target text.

[0206] In this embodiment, the target text to be synthesized is received; the target phoneme sequence corresponding to the target text is input into the speech synthesis model to obtain the target speech corresponding to the target text output by the speech synthesis model; the speech synthesis model is trained using a posterior encoder, a bypass guidance module, training recordings, and training text; the bypass guidance module is used to generate guidance information; the guidance information is used to guide the posterior encoder so that the posterior encoder extracts the backbone speech information in the linear spectrum corresponding to the training recording, and the backbone speech information is used to train the speech synthesis model. This embodiment, by configuring a bypass guidance module for the posterior encoder, realizes the use of guidance information output by the bypass guidance module to specifically guide the posterior encoder to capture the backbone information in the linear spectrum corresponding to the training recording. This embodiment helps improve the overall training effect of the speech synthesis model, effectively solves the problems of inaccurate pronunciation and fluctuating sound quality of the synthesized target speech, enhances the clarity and naturalness of the target speech, and thus significantly improves the user experience.

[0207] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0208] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0209] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 6 An exemplary apparatus 600 is schematically shown that can be used to implement the various embodiments described in this application.

[0210] In one embodiment, Figure 6 An exemplary device 600 is shown, which includes one or more processors 602, a control module (chipset) 604 coupled to at least one of the processors 602, a memory 606 coupled to the control module 604, a non-volatile memory (NVM) / storage device 608 coupled to the control module 604, one or more input / output devices 610 coupled to the control module 604, and a network interface 612 coupled to the control module 604.

[0211] Processor 602 may include one or more single-core or multi-core processors, and processor 602 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 600 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0212] In some embodiments, the apparatus 600 may include one or more computer-readable media (e.g., memory 606 or NVM / storage device 608) having instructions 614 and one or more processors 602 that are combined with the one or more computer-readable media and configured to execute the instructions 614 to implement the module and thus perform the actions described in this disclosure.

[0213] In one embodiment, the control module 604 may include any suitable interface controller to provide any suitable interface to at least one of the processors 602 and / or any suitable device or component communicating with the control module 604.

[0214] The control module 604 may include a memory controller module to provide an interface to the memory 606. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0215] Memory 606 may be used, for example, to load and store data and / or instructions 614 for device 600. In one embodiment, memory 606 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 606 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0216] In one embodiment, the control module 604 may include one or more input / output controllers to provide an interface to the NVM / storage device 608 and (one or more) input / output devices 610.

[0217] For example, NVM / storage device 608 may be used to store data and / or instructions 614. NVM / storage device 608 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0218] NVM / storage device 608 may include storage resources that are physically part of a device on which device 600 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 608 may be accessed via a network through one or more input / output devices 610.

[0219] One or more input / output devices 610 may provide an interface for device 600 to communicate with any other suitable device. Input / output devices 610 may include communication components, audio components, sensor components, etc. A network interface 612 may provide an interface for device 600 to communicate via one or more networks. Device 600 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0220] In one embodiment, at least one of the processors 602 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 604. In one embodiment, at least one of the processors 602 may be logically packaged with one or more controllers of the control module 604 to form a system-in-package (SiP). In one embodiment, at least one of the processors 602 may be integrated with the logic of one or more controllers of the control module 604 on the same die. In one embodiment, at least one of the processors 602 may be integrated with the logic of one or more controllers of the control module 604 on the same die to form a system-on-a-chip (SoC).

[0221] In various embodiments, device 600 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop, handheld computing device, tablet, netbook, etc.). In various embodiments, device 600 may have more or fewer components and / or different architectures. For example, in some embodiments, device 600 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0222] The detection device may use a main control chip as a processor or control module, and sensor data, position information, etc. may be stored in a memory or NVM / storage device. The sensor group may be used as an input / output device, and the communication interface may include a network interface.

[0223] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0224] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0225] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable speech synthesis terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable speech synthesis terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0226] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable speech synthesis terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0227] These computer program instructions can also be loaded onto a computer or other programmable speech synthesis terminal device, causing a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0228] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0229] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0230] The above provides a detailed description of a speech synthesis method and apparatus, an electronic device, and a storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech synthesis method characterized by, The method comprises: receiving target text to be synthesized speech; inputting a target phoneme sequence corresponding to the target text into a speech synthesis model to obtain target speech corresponding to the target text output by the speech synthesis model, wherein the speech synthesis model comprises a decoder, a flow module, a text encoder, a projection layer, and a random duration predictor; the speech synthesis model is trained using the decoder, the flow module, the text encoder, the projection layer, the random duration predictor, a posteriori encoder, a bypass guide module, training audio, and training text; the bypass guide module is used to generate guide information; the guide information is used to guide the posteriori encoder to extract core speech information in a linear spectrum corresponding to the training audio, and the core speech information is used to train the speech synthesis model; wherein the bypass guide module is composed of a plurality of coupled layers connected in series, and the training process of the speech synthesis model comprises: inputting pronunciation information corresponding to the training audio into the bypass guide module; when the pronunciation information passes through the coupled layers in the bypass guide module one by one, obtaining the guide information output by each coupled layer, and obtaining a self-supervised loss output by the last coupled layer; inputting the guide information and the linear spectrum corresponding to the training audio into the posteriori encoder to obtain training speech mean and training speech variance; randomly sampling a Gaussian distribution model corresponding to the training speech mean and the training speech variance to obtain a speech sampling result; inputting the speech sampling result into the decoder to obtain a speech generation loss; inputting the speech sampling result into the flow module to obtain a training intermediate vector; inputting a phoneme sequence corresponding to the training text into the text encoder to obtain a training text matrix; inputting the training text matrix into the projection layer to obtain training text mean and training text variance; using the training intermediate vector to perform timing alignment on the training text mean and the training text variance to obtain a first training alignment parameter; inputting the training text matrix into the random duration predictor to obtain a second training alignment parameter; comparing the first training alignment parameter and the second training alignment parameter to obtain a duration loss; comparing a training text Gaussian distribution shaped by the training text mean and the training text variance with a training speech Gaussian distribution shaped by the training speech mean and the training speech variance to obtain relative entropy; updating neural network parameters of the speech synthesis model according to the self-supervised loss, the speech generation loss, the duration loss, and the relative entropy.

2. The method of claim 1, wherein, The posteriori encoder is composed of a plurality of convolutional layers connected in series, the number of the convolutional layers is the same as the number of the coupled layers in the bypass guide module, each convolutional layer corresponds to each coupled layer one by one, and the inputting of the guide information and the linear spectrum corresponding to the training audio into the posteriori encoder comprises: If the current convolutional layer is the first convolutional layer in the posterior encoder, the guide information output by the coupling layer corresponding to the current convolutional layer and the linear spectrum corresponding to the training recording are input into the current convolutional layer; If the current convolutional layer is not the first convolutional layer in the posterior encoder, the guide information output by the coupling layer corresponding to the current convolutional layer and the output information of the previous convolutional layer are input into the current convolutional layer.

3. The method of claim 2, wherein, The method further comprises: generating a first guide coefficient corresponding to the current convolutional layer according to the guide information output by the coupling layer corresponding to the current convolutional layer; calculating a second guide coefficient corresponding to the linear spectrum of the training recording or the output information of the previous convolutional layer; calculating according to the first guide coefficient and the second guide coefficient to obtain the output of the current convolutional layer; If the current convolutional layer is the last convolutional layer in the posterior encoder, the training speech mean and the training speech variance are obtained according to the output of the current convolutional layer.

4. The method of claim 1, wherein, The method further comprises: inputting the target phoneme sequence corresponding to the target text into the text encoder to obtain a text matrix corresponding to the target phoneme sequence; inputting the text matrix into the projection layer to obtain a text mean and a text variance; randomly sampling a text Gaussian distribution corresponding to the text mean and the text variance to obtain a text sampling result; inputting the text matrix into the random duration predictor to obtain an alignment parameter; performing time copying on the text sampling result according to the alignment parameter to obtain a target text sampling result; inputting the target text sampling result into the flow module to obtain an intermediate vector; inputting the intermediate vector into the decoder to output the target speech corresponding to the target text.

5. A speech synthesis apparatus characterized by comprising: The apparatus comprises: a text receiving module configured to receive a target text to be synthesized into speech; a speech synthesis module configured to input a target phoneme sequence corresponding to the target text into a speech synthesis model to obtain a target speech corresponding to the target text output by the speech synthesis model, wherein the speech synthesis model comprises a decoder, a flow module, a text encoder, a projection layer, and a random duration predictor; the speech synthesis model is trained using the decoder, the flow module, the text encoder, the projection layer, the random duration predictor, a posterior encoder, a bypass guide module, a training recording, and a training text; the bypass guide module is configured to generate guide information; the guide information is used to guide the posterior encoder to extract core speech information in a linear spectrum corresponding to the training recording, and the core speech information is used to train the speech synthesis model. The bypass guide module comprises a plurality of coupling layers connected in series, and the apparatus comprises: a training speech input module configured to input pronunciation information corresponding to the training recording into the bypass guide module. The guiding information acquisition module is configured to acquire the guiding information output by each of the coupling layers when the pronunciation information passes through the coupling layers in the bypass guiding module one by one, and acquire a self-supervised loss output by a last coupling layer. The normalization guiding module is configured to input the guiding information into the posterior encoder together with a linear spectrum corresponding to the training audio to obtain a training speech mean value and a training speech variance. The random sampling submodule is configured to randomly sample a Gaussian distribution model corresponding to the training speech mean value and the training speech variance to obtain a speech sampling result. The decoder processing submodule is configured to input the speech sampling result into the decoder to obtain a speech generation loss. The normalized flow processing submodule is configured to input the speech sampling result into the flow module to obtain a training intermediate vector. The text processing submodule is configured to input a phoneme sequence corresponding to the training text into the text encoder to obtain a training text matrix, input the training text matrix into the projection layer to obtain a training text mean value and a training text variance, perform timing alignment on the training text mean value and the training text variance by using the training intermediate vector to obtain a first training alignment parameter, input the training text matrix into the random duration predictor to obtain a second training alignment parameter, compare the first training alignment parameter and the second training alignment parameter to obtain a duration loss, and compare a training text Gaussian distribution shaped by the training text mean value and the training text variance with a training speech Gaussian distribution shaped by the training speech mean value and the training speech variance to obtain a relative entropy. The parameter updating submodule is configured to update neural network parameters of the speech synthesis model according to the self-supervised loss, the speech generation loss, the duration loss, and the relative entropy.

6. An electronic device, comprising: comprise: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the speech synthesis method of any one or more of claims 1-4.

7. One or more machine-readable media having stored thereon executable code that, when executed by a processor, causes the processor to perform the speech synthesis method of any one or more of claims 1-4.

Images