Speech synthesis method, training method, apparatus, device, and medium

Abstract

The speech synthesis method, the training method, the device, the equipment and the medium provided by the embodiment of the application comprise: obtaining text feature data of a text sequence to be predicted; inputting the text feature data into an alignment position prediction model to output alignment position prediction data; inputting the alignment position prediction data into a reconstruction mapping model to output spectrum feature prediction data; in the above manner, the trained alignment position prediction model can directly obtain alignment position prediction data according to the text feature data, and can solve the technical problem of poor speech synthesis effect caused by the difficulty of aligning the text feature and the spectrum feature in the prior art, which is conducive to improving the effect of speech synthesis. After obtaining the alignment position prediction data, the trained reconstruction mapping model can directly obtain spectrum feature prediction data according to the alignment position prediction data, and the entire speech synthesis stage does not need to rely on spectrum data, which is conducive to improving the speed of speech synthesis.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, specifically to a speech synthesis method, training method, device, equipment, and medium. Background Technology

[0002] TTS (text-to-speech) is a text-to-speech synthesis algorithm. Currently, mainstream speech synthesis tasks are based on neural networks (Neural TTS). Compared to traditional concatenative and parametric speech synthesis models, Neural TTS produces smoother, more natural-sounding speech.

[0003] In existing technologies, most text-to-speech synthesis algorithms rely on reversible networks. The affine coupling layer in reversible networks is inefficient, with only half of the parameters being updated in each calculation, resulting in slow training speed. In addition, the speech synthesis effect of existing text-to-speech synthesis algorithms needs to be improved. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a speech synthesis method, training method, apparatus, device, and medium to solve the above technical problems.

[0005] In a first aspect, embodiments of this application provide a speech synthesis method, including:

[0006] Obtain the text feature data of the text sequence to be predicted;

[0007] The text feature data is input into a pre-trained alignment position prediction model, and the alignment position prediction data corresponding to the text feature data is output. The alignment position prediction model is trained based on real text sequences and real linear spectra obtained from real speech data.

[0008] The alignment position prediction data is input into a pre-trained reconstruction mapping model, and the corresponding spectral feature prediction data is output. The reconstruction mapping model is trained based on real text sequences and real linear spectra obtained from real speech data.

[0009] Optionally, the training steps of the alignment position prediction model include:

[0010] At least one training sample is obtained, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being obtained based on real speech data;

[0011] The text sequence is input into a text encoder, which outputs the text feature data of the text sequence.

[0012] The linear spectrum is input into a variational autoencoder, which outputs the spectral feature data of the linear spectrum.

[0013] The alignment position calculation data of the training samples is obtained based on the text feature data and the spectral feature data;

[0014] The text feature data is input into the alignment position prediction model, and the alignment position prediction data corresponding to the text feature data is output.

[0015] The prediction error is calculated based on the alignment position calculation data and the alignment position prediction data. The parameters of the alignment position prediction model are adjusted based on the prediction error until the alignment position prediction model reaches the training convergence condition.

[0016] Optionally, the training steps of the reconstructed mapping model include:

[0017] At least one training sample is obtained, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being obtained based on real speech data;

[0018] The text sequence is input into a text encoder, which outputs the text feature data of the text sequence.

[0019] The linear spectrum is input into a variational autoencoder, which outputs the spectral feature data of the linear spectrum.

[0020] The alignment position calculation data of the training samples is obtained based on the text feature data and the spectral feature data;

[0021] The alignment position calculation data is input into the reconstruction mapping model, and the spectral feature prediction data corresponding to the alignment position calculation data is output.

[0022] Obtain the KL divergence of the spectral feature data and the spectral feature prediction data, and adjust the parameters of the reconstructed mapping model according to the KL divergence until the reconstructed mapping model reaches the training convergence condition.

[0023] Optionally, the step of obtaining the alignment position calculation data of the training samples based on the text feature data and the spectral feature data includes:

[0024] The first alignment matrix of the text feature data is obtained based on the text feature data and the spectral feature data;

[0025] The index mapping vector of the spectral feature data is obtained based on the first alignment matrix and the text feature data;

[0026] The second alignment matrix of the spectral feature data is obtained based on the index mapping vector and the text feature data;

[0027] The alignment position calculation data of the text feature data is obtained based on the second alignment matrix and the spectral feature data.

[0028] Optionally, after inputting the alignment position prediction data into a pre-trained reconstruction mapping model and outputting the spectral feature prediction data corresponding to the alignment position prediction data, the method further includes:

[0029] The spectral feature prediction data is input into the speech generator, which outputs the corresponding speech synthesis data.

[0030] The training steps for the speech generator include:

[0031] Obtain at least one training sample, the training sample comprising real speech data and a linear spectrum extracted from the real speech data;

[0032] The linear spectrum is input into a variational autoencoder, which outputs the spectral feature data of the linear spectrum.

[0033] The spectral feature data is input into the speech generator, which outputs synthesized speech data.

[0034] The synthesis error is calculated based on the real speech data and the synthesized speech data. The parameters of the speech generator are then adjusted based on the synthesis error until the speech generator reaches the training convergence condition.

[0035] Secondly, embodiments of this application also provide a training method for a speech synthesis model, the speech synthesis model including an alignment position prediction model and a reconstruction mapping model, the training method including:

[0036] At least one training sample is obtained, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being obtained based on real speech data;

[0037] The text sequence is input into a text encoder, which outputs the text feature data of the text sequence.

[0038] The linear spectrum is input into a variational autoencoder, which outputs the spectral feature data of the linear spectrum.

[0039] The alignment position calculation data of the training samples is obtained based on the text feature data and the spectral feature data;

[0040] The text feature data is input into the alignment position prediction model, and the alignment position prediction data corresponding to the text feature data is output.

[0041] The prediction error is calculated based on the alignment position calculation data and the alignment position prediction data. The parameters of the alignment position prediction model are adjusted based on the prediction error until the alignment position prediction model reaches the training convergence condition.

[0042] The alignment position calculation data is input into the reconstruction mapping model, and the spectral feature prediction data corresponding to the alignment position calculation data is output.

[0043] Obtain the KL divergence of the spectral feature data and the spectral feature prediction data, and adjust the parameters of the reconstructed mapping model according to the KL divergence until the reconstructed mapping model reaches the training convergence condition.

[0044] Optionally, the speech synthesis model further includes a speech generator, and the training samples further include the real speech data;

[0045] The training method also includes:

[0046] The spectral feature data is input into the speech generator, which outputs synthesized speech data.

[0047] The synthesis error is calculated based on the real speech data and the synthesized speech data. The parameters of the speech generator are then adjusted based on the synthesis error until the speech generator reaches the training convergence condition.

[0048] Thirdly, embodiments of this application also provide a speech synthesis device, including:

[0049] The text feature extraction module is used to obtain the text feature data of the text sequence to be predicted;

[0050] The alignment position prediction module is used to input the text feature data into a pre-trained alignment position prediction model and output the alignment position prediction data corresponding to the text feature data. The alignment position prediction model is trained based on real text sequences and real linear spectra obtained from real speech data.

[0051] The mapping module is used to input the alignment position prediction data into a pre-trained reconstruction mapping model and output the spectral feature prediction data corresponding to the alignment position prediction data. The reconstruction mapping model is trained based on real text sequences and real linear spectra obtained from real speech data.

[0052] Fourthly, embodiments of this application also provide an electronic device, including a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; when the processor executes the program instructions stored in the memory, it implements the above-described speech synthesis method or the above-described speech synthesis model training method.

[0053] Fifthly, embodiments of this application also provide a storage medium storing program instructions, which, when executed by a processor, implement the above-described speech synthesis method or the above-described training method for the speech synthesis model.

[0054] The speech synthesis method, training method, apparatus, device, and medium provided in this application include: acquiring text feature data of a text sequence to be predicted; inputting the text feature data into a pre-trained alignment position prediction model and outputting alignment position prediction data corresponding to the text feature data, wherein the alignment position prediction model is trained based on a real text sequence and a real linear spectrum obtained from real speech data; inputting the alignment position prediction data into a pre-trained reconstruction mapping model and outputting spectral feature prediction data corresponding to the alignment position prediction data, wherein the reconstruction mapping model is trained based on a real text sequence and a real linear spectrum obtained from real speech data. Through the above method, the alignment position prediction data can be directly obtained from the text feature data using the trained alignment position prediction model, which can solve the technical problem of poor speech synthesis effect caused by the difficulty in aligning text features and spectral features in the prior art, thus improving the effect of speech synthesis. After obtaining the alignment position prediction data, the spectral feature prediction data can be directly obtained from the alignment position prediction data using the trained reconstruction mapping model, and the entire speech synthesis stage does not rely on spectral data, which is beneficial to improving the speed of speech synthesis.

[0055] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A schematic flowchart of a speech synthesis method provided in an embodiment of this application is shown.

[0058] Figure 2This diagram illustrates the principle of the speech synthesis model during the model training phase in a speech synthesis method provided in an embodiment of this application.

[0059] Figure 3 A flowchart illustrating a training method for a speech synthesis model provided in an embodiment of this application is shown.

[0060] Figure 4 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of this application is shown.

[0061] Figure 5 A schematic diagram of the structure of a training device for a speech synthesis model provided in an embodiment of this application is shown.

[0062] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0063] Figure 7 A schematic diagram of the structure of a storage medium provided in an embodiment of this application is shown;

[0064] Figure 8 This diagram illustrates the principle of the speech synthesis model in the speech synthesis stage of a speech synthesis method provided in an embodiment of this application. Detailed Implementation

[0065] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0066] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0067] In the embodiments of this application, "at least one" refers to one or more; "multiple" refers to two or more. In the description of this application, the terms "first," "second," "third," etc., are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0068] References such as “one embodiment” or “some embodiments” as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the terms “comprising,” “including,” “having,” and variations thereof, as used in this specification, mean “including, but not limited to,” unless otherwise specifically emphasized.

[0069] It should be noted that in the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that there can be three kinds of relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0070] It should be noted that in the embodiments of this application, "connection" can be understood as electrical connection. The connection between two electrical components can be a direct or indirect connection between the two electrical components. For example, the connection between A and B can be a direct connection between A and B, or an indirect connection between A and B through one or more other electrical components.

[0071] One embodiment of this application provides a speech synthesis method. The execution entity of the speech synthesis method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices that can be configured to execute the speech synthesis method provided in this embodiment. In other words, the speech synthesis method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0072] Please see Figure 1 The diagram shown is a flowchart illustrating a speech synthesis method according to an embodiment of this application. It should be noted that if substantially the same result is achieved, the method of this application does not necessarily require further elaboration. Figure 1 The illustrated process sequence is limited. In this embodiment, please also refer to... Figure 8 The diagram shows the principle of the speech synthesis model in the speech synthesis stage. The speech synthesis method includes the following steps:

[0073] S10, Obtain the text feature data of the text sequence to be predicted;

[0074] The text sequence to be predicted is obtained from the text to be synthesized. The text to be synthesized is in written language form, usually a phrase, a sentence, or a combination of sentences with a complete meaning. For example, the text to be synthesized could be "Hello, Ping An Insurance!".

[0075] In an optional implementation, the text sequence can be a sequence of Chinese characters in the text to be synthesized.

[0076] In another optional embodiment, the text sequence may be the phoneme sequence of the text to be synthesized, that is, the phoneme character sequence of the text to be synthesized. Among them, a phoneme is the smallest unit in speech and can be analyzed based on the pronunciation actions in the syllables of a word. One action constitutes one phoneme. For example, in Chinese, there are 32 phonemes, which can be divided into initials and finals. For example, for the text to be synthesized, its corresponding phoneme sequence can be composed of the initials and finals of each character in turn. Taking the text to be synthesized as "你好中国平安" as an example, its corresponding phoneme sequence can be "n, i, h, ao, zh, ong, g, uo, p, ing, an". It should be understood that its corresponding phoneme sequence can also be in other forms, and the present application does not specifically limit the form of the phoneme sequence.

[0077] Among them, the text feature data may be character feature data generated according to the text sequence to be predicted. The text feature data is generated according to the character feature vectors corresponding to each character in the text sequence to be predicted. For example, the character feature vector can be a vector obtained by one-hot encoding or other encoding methods for the corresponding character.

[0078] S20. Input the text feature data into a pre-trained alignment position prediction model, and output the alignment position prediction data corresponding to the text feature data. Among them, the alignment position prediction model is trained according to the real text sequence and real linear spectrum obtained from real speech data;

[0079] Among them, the alignment position prediction model can be a neural network, which is used to predict the alignment position according to the input text feature data. For the convenience of description, the training process of the alignment position prediction model will be introduced first. Please refer to Figure 2 as shown, which specifically includes the following steps:

[0080] S21. Obtain at least one training sample, where the training sample includes a text sequence and a linear spectrum corresponding to the text sequence. The text sequence and the linear spectrum are obtained according to real speech data;

[0081] Among them, real speech data can be subjected to speech recognition to extract the text content, and then the text sequence can be generated according to the text content. For the description of the text sequence, please refer to the above, and details will not be repeated here.

[0082] Among them, real speech data can be subjected to short-time Fourier transform (short-time Fourier transform, or short-term Fourier transform, STFT) to extract the linear spectrum (Linear Spectrogram).

[0083] S22, the text sequence is input to the text encoder, and the text feature data of the text sequence is output;

[0084] The text encoder maps each character in the text sequence to a corresponding character feature vector, and the character feature vectors form text feature data p[p0, p1, p2, ..., p]. i , ..., p T1-1 ], where 0≤i≤T1-1, T1 is the number of vectors in the text feature data, and T1 is an integer greater than or equal to 2.

[0085] S23, input the linear spectrum to the variational autoencoder (Spec encoder) and output the spectral feature data of the linear spectrum;

[0086] The variational autoencoder extracts features from the input linear spectrum and outputs spectral feature data of the linear spectrum, which is q[q0, q1, q2, ..., q]. j , ..., q T2-1 ], where 0≤j≤T2-1, T2 is the number of vectors in the spectral feature data, and T2 is an integer greater than or equal to 2. The spectral feature data is the mean and variance of a multivariate Gaussian distribution.

[0087] S24, Obtain the alignment position calculation data of the training sample based on the text feature data and the spectral feature data;

[0088] As one implementation method, the alignment position calculation data is obtained as follows:

[0089] S241, Obtain the first alignment matrix of the text feature data based on the text feature data and the spectral feature data;

[0090] As one implementation method, cross-attention is performed on text feature data and spectral feature data to obtain the first alignment matrix. Specifically, the first alignment matrix α is calculated as follows:

[0091]

[0092] Where, α i，j p is the matrix element in the i-th row and j-th column of the first alignment matrix α. i Let q be the i-th feature vector in the text feature data. j Let p be the j-th feature vector in the spectral feature data. mLet be the m-th feature vector in the text feature data, D be the dimension of the outputs of the text encoder and variational autoencoder, exp() be the exponential function with the natural constant e as the base, and T1 be the length of the text feature data. The spectral feature data q can be calculated based on the first alignment matrix α and the text feature data p.

[0093] S242, obtain the index mapping vector (IMV) of the spectral feature data based on the first alignment matrix and the text feature data;

[0094] As one implementation method, the index mapping vector π is calculated as follows:

[0095]

[0096] Where, π j Let α be the index mapping vector of the j-th index. i，j p is the matrix element in the i-th row and j-th column of the first alignment matrix α. i Let be the i-th feature vector in the text feature data.

[0097] S243, Obtain the second alignment matrix of the spectral feature data based on the index mapping vector and the text feature data;

[0098] As one implementation method, the second alignment matrix γ is calculated as follows:

[0099]

[0100] Where, γ i，j p is the matrix element in the i-th row and j-th column of the second alignment matrix γ. i Let π be the i-th feature vector in the text feature data. j For the j-th index mapping vector, π n Let be the nth index mapping vector, δ be the alignment variance, exp() be an exponential function with base e, and T2 be the length of the spectral feature data. The text feature data p can be calculated based on the second alignment matrix γ and the spectral feature data q.

[0101] S244, Obtain the alignment position calculation data of the text feature data based on the second alignment matrix and the spectral feature data;

[0102] As one implementation method, the alignment position calculation data e is calculated as follows:

[0103]

[0104] Among them, e i For the i-th alignment position data, γi，m Let q be the matrix element in the i-th row and n-th column of the second alignment matrix γ. n This is the nth feature vector in the spectral feature data.

[0105] S25, input the text feature data into the aligned position predictor and output the aligned position prediction data corresponding to the text feature data;

[0106] Among them, the alignment position prediction model is used to predict the alignment position calculation data e above based on the text feature data p, and the alignment position prediction model outputs the alignment position prediction data e′.

[0107] S26, calculate the prediction error based on the alignment position calculation data and the alignment position prediction data, and adjust the parameters of the alignment position prediction model based on the prediction error until the alignment position prediction model reaches the training convergence condition.

[0108] In the training process of the alignment position prediction model, it is desirable for the alignment position prediction data e′ to be as close as possible to the alignment position calculation data e. The alignment position calculation data e is used as the true value, and the alignment position prediction data e′ is used as the predicted value. The prediction error can be calculated based on the difference between the two to adjust the parameters of the alignment position prediction model until the alignment position prediction model reaches the training convergence condition. The alignment position prediction model that has reached the training convergence condition is used as the trained alignment position prediction model and used for the prediction of alignment position in actual speech synthesis.

[0109] In this implementation, during the training process of the alignment position prediction model, the alignment position calculation data e is finally obtained based on the text sequence and linear spectrum of real speech data. The text feature data p is then input into the alignment position prediction model to output the alignment position prediction data e′, thereby improving the training speed and prediction accuracy of the alignment position prediction model.

[0110] S30, the alignment position prediction data is input into a pre-trained reconstruction mapping model, and the spectral feature prediction data corresponding to the alignment position prediction data is output. The reconstruction mapping model is trained based on real text sequences and real linear spectra obtained from real speech data.

[0111] The reconstruction mapping model includes an alignment reconstruction module and a decoder. The alignment reconstruction module is used to reconstruct the first alignment matrix α based on the alignment position prediction data e′, and the decoder is used to obtain spectral feature prediction data based on the reconstructed first alignment matrix.

[0112] In one implementation, the alignment position prediction data e′ is input into the alignment reconstruction module, which outputs a reconstructed first alignment matrix α′; the reconstructed first alignment matrix α′ is then input into the decoder, which outputs spectral feature prediction data q′. Specifically, the calculation method for the reconstructed first alignment matrix α′ is as follows:

[0113]

[0114] Where, α′ i，j To reconstruct the matrix elements in the i-th row and j-th column of the first alignment matrix α′, q j Let e′ be the j-th eigenvector in the spectral feature data. i Let e′ be the predicted vector for the i-th alignment position. m Let be the m-th alignment position prediction vector, δ be the alignment variance, exp() be an exponential function with the natural constant e as the base, and T1 be the length of the text feature data, which is also the length of the alignment position prediction data. As one implementation, the decoder can employ a neural network, and the spectral feature prediction data output by the decoder is also the mean and variance of a multivariate Gaussian distribution.

[0115] As one implementation method, the training steps of the reconstructed mapping model include:

[0116] S31, Obtain at least one training sample, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being obtained based on real speech data;

[0117] S32, input the text sequence into the text encoder and output the text feature data of the text sequence;

[0118] S33, the linear spectrum is input to the variational autoencoder, and the spectral feature data of the linear spectrum is output;

[0119] S34, Obtain the alignment position calculation data of the training sample based on the text feature data and the spectral feature data;

[0120] For details of steps S31 to S34, please refer to the description of steps S21 to S24, which will not be repeated here.

[0121] S35, input the alignment position calculation data into the reconstruction mapping model, and output the spectral feature prediction data corresponding to the alignment position calculation data;

[0122] During the training of the reconstruction mapping model, the input is the alignment position calculation data *e*; while during speech synthesis, the input to the reconstruction mapping model is the alignment position prediction data *e′*. During training, the reconstruction mapping model calculates the first alignment matrix α′ according to the following formula:

[0123]

[0124] Where, α′ i，j To reconstruct the matrix elements in the i-th row and j-th column of the first alignment matrix α′, q j Let e ​​be the j-th feature vector in the spectral feature data. i Let e ​​be the i-th alignment position vector. m Let be the m-th alignment position vector, δ be the alignment variance, exp() be an exponential function with the natural constant e as the base, and T1 be the length of the text feature data, which is also the length of the alignment position calculation data. As one implementation, the decoder can employ a neural network, and the decoder output spectral feature prediction data is also the mean and variance of a multivariate Gaussian distribution.

[0125] S36, Obtain the KL (Kullback-Leibler) divergence of the spectral feature data and the spectral feature prediction data, and adjust the parameters of the reconstruction mapping model according to the KL divergence until the reconstruction mapping model reaches the training convergence condition;

[0126] In the training process of the reconstruction mapping model, it is desirable for the spectral feature prediction data q′ to be as close as possible to the spectral feature data q. The spectral feature data q is used as the true value, and the spectral feature prediction data q′ is used as the predicted value. The KL divergence between the two can be used to characterize the prediction error and adjust the parameters of the reconstruction mapping model until the reconstruction mapping model reaches the training convergence condition. The reconstruction mapping model that has reached the training convergence condition is used as the trained reconstruction mapping model and used for the prediction of spectral feature data in actual speech synthesis.

[0127] In this implementation, during the training process of the reconstruction mapping model, the spectral feature data q and the alignment position calculation data e are finally obtained based on the text sequence and linear spectrum of real speech data. The alignment position calculation data e is then input into the alignment position prediction model to output the spectral feature prediction data q′, thereby improving the training speed and prediction accuracy of the reconstruction mapping model.

[0128] The speech synthesis method in this embodiment can directly obtain alignment position prediction data based on text feature data by using a trained alignment position prediction model. This can solve the technical problem of poor speech synthesis effect caused by the difficulty in aligning text features and spectral features in the prior art, and is conducive to improving the effect of speech synthesis. After obtaining the alignment position prediction data, the trained reconstruction mapping model can directly obtain spectral feature prediction data based on the alignment position prediction data. The entire speech synthesis stage does not need to rely on spectral data, which is conducive to improving the speed of speech synthesis.

[0129] As one implementation method, the speech synthesis method of this embodiment further includes, after step S30:

[0130] S40, the spectral feature prediction data is input into the speech generator, and the corresponding speech synthesis data is output.

[0131] Speech synthesis data can be obtained through a speech generator (Hifi-GAN generator). Please refer to [link / reference]. Figure 8 As shown, the spectral feature prediction data output by the reconstruction mapping model is input into the speech generator to obtain speech synthesis data.

[0132] Please continue reading. Figure 2 As shown, when training the speech generator, the linear spectrum extracted from real speech data is used. The spectral feature data of the linear spectrum output by the variational autoencoder (Spec encoder) is used as the input of the speech generator. The speech generator has strong prior knowledge, and the entire model training is much easier. It can be trained end-to-end directly without any pre-training.

[0133] As one implementation method, the training steps of the speech generator include:

[0134] S41, Obtain at least one training sample, the training sample including real speech data and a linear spectrum extracted from the real speech data;

[0135] S42, the linear spectrum is input to the variational autoencoder, and the spectral feature data of the linear spectrum is output;

[0136] S43, input the spectral feature data into the speech generator and output synthesized speech data;

[0137] S44, calculate the synthesis error based on the real speech data and the synthesized speech data, and adjust the parameters of the speech generator based on the synthesis error until the speech generator reaches the training convergence condition.

[0138] In this embodiment, the inclusion of a variational autoencoder (Spec encoder) makes training the Hifi-GAN generator easier. Without a Spec encoder, during model training, only the spectral feature prediction data output by the reconstruction mapping model could be input to the speech generator to output synthesized speech data. Since this spectral feature prediction data is obtained from text sequence prediction, the information it contains as input to the speech generator is limited to textual information, which is detrimental to training and improving the quality of speech synthesis. With the Spec encoder, during model training, the spectral feature data output by the Spec encoder is input to the speech generator to output synthesized speech data. This spectral feature data originates from a linear spectrum extracted from real speech data, providing relatively complete spectral information. As input to the speech generator, the spectral feature data includes at least pitch variation and energy variation information, making training easier and improving the quality of speech synthesis.

[0139] Please see Figure 3 The diagram shown is a flowchart illustrating a training method for a speech synthesis model according to an embodiment of this application. It should be noted that if substantially the same result is obtained, the method of this application does not necessarily require further elaboration. Figure 3 The illustrated process order is limited. In this embodiment, the speech synthesis model includes an alignment position prediction model and a reconstruction mapping model, and the training method of the speech synthesis model includes the following steps:

[0140] S51, Obtain at least one training sample, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being obtained based on real speech data;

[0141] S52, input the text sequence into the text encoder and output the text feature data of the text sequence;

[0142] S53, the linear spectrum is input to the variational autoencoder, and the spectral feature data of the linear spectrum is output;

[0143] S54, Obtain the alignment position calculation data of the training sample based on the text feature data and the spectral feature data;

[0144] S55, input the text feature data into the alignment position prediction model, and output the alignment position prediction data corresponding to the text feature data;

[0145] S56, calculate the prediction error based on the alignment position calculation data and the alignment position prediction data, and adjust the parameters of the alignment position prediction model based on the prediction error until the alignment position prediction model reaches the training convergence condition.

[0146] S57, input the alignment position calculation data into the reconstruction mapping model, and output the spectral feature prediction data corresponding to the alignment position calculation data;

[0147] S58, obtain the KL divergence of the spectral feature data and the spectral feature prediction data, and adjust the parameters of the reconstruction mapping model according to the KL divergence until the reconstruction mapping model reaches the training convergence condition.

[0148] For details on each step of the training method in this embodiment, please refer to the description of the above embodiment.

[0149] In one implementation, the speech synthesis model further includes a speech generator, and the training samples further include the real speech data.

[0150] The training method also includes:

[0151] S61, the spectral feature data is input into the speech generator, and synthesized speech data is output;

[0152] S62, calculate the synthesis error based on the real speech data and the synthesized speech data, and adjust the parameters of the speech generator based on the synthesis error until the speech generator reaches the training convergence condition.

[0153] The specific steps of the training method in this embodiment can be found in the description of the above embodiments.

[0154] like Figure 4 As shown in the figure, one embodiment of this application provides a speech synthesis device 40, which includes: a text feature extraction module 41, an alignment position prediction module 42, and a mapping module 43. The text feature extraction module 41 is used to acquire text feature data of a text sequence to be predicted. The alignment position prediction module 42 is used to input the text feature data into a pre-trained alignment position prediction model and output alignment position prediction data corresponding to the text feature data. The alignment position prediction model is trained based on a real text sequence and a real linear spectrum obtained from real speech data. The mapping module 43 is used to input the alignment position prediction data into a pre-trained reconstruction mapping model and output spectral feature prediction data corresponding to the alignment position prediction data. The reconstruction mapping model is trained based on a real text sequence and a real linear spectrum obtained from real speech data.

[0155] In one implementation, the alignment position prediction module 42 is further configured to: acquire at least one training sample, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being acquired based on real speech data; input the text sequence to a text encoder and output text feature data of the text sequence; input the linear spectrum to a variational autoencoder and output spectral feature data of the linear spectrum; acquire alignment position calculation data of the training sample based on the text feature data and the spectral feature data; input the text feature data into the alignment position prediction model and output alignment position prediction data corresponding to the text feature data; calculate the prediction error based on the alignment position calculation data and the alignment position prediction data; and adjust the parameters of the alignment position prediction model based on the prediction error until the alignment position prediction model reaches the training convergence condition.

[0156] In one implementation, the mapping module 43 is further configured to: acquire at least one training sample, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being acquired based on real speech data; input the text sequence to a text encoder and output text feature data of the text sequence; input the linear spectrum to a variational autoencoder and output spectral feature data of the linear spectrum; acquire alignment position calculation data of the training sample based on the text feature data and the spectral feature data; input the alignment position calculation data to the reconstruction mapping model and output spectral feature prediction data corresponding to the alignment position calculation data; acquire the KL divergence of the spectral feature data and the spectral feature prediction data, and adjust the parameters of the reconstruction mapping model based on the KL divergence until the reconstruction mapping model reaches the training convergence condition.

[0157] In one implementation, the alignment position prediction module 42 and the mapping module 43 are further configured to: obtain a first alignment matrix of the text feature data based on the text feature data and the spectral feature data; obtain an index mapping vector of the spectral feature data based on the first alignment matrix and the text feature data; obtain a second alignment matrix of the spectral feature data based on the index mapping vector and the text feature data; and obtain alignment position calculation data of the text feature data based on the second alignment matrix and the spectral feature data.

[0158] As one implementation, the speech synthesis device 40 of this embodiment further includes: a speech generation module, used to obtain a corresponding predicted linear spectrum based on the spectral feature prediction data; and to obtain speech synthesis data based on the text sequence and the predicted linear spectrum.

[0159] like Figure 5 As shown, one embodiment of this application provides a training device for a speech synthesis model. The speech synthesis model includes an alignment position prediction model and a reconstruction mapping model. The training device 50 for the speech synthesis model includes: a sample extraction module 51, a text feature module 52, a spectrum feature module 53, an alignment position calculation module 54, a first training module 55, and a second training module 56. The sample extraction module 51 is used to acquire at least one training sample, which includes a text sequence and a linear spectrum corresponding to the text sequence. The text sequence and the linear spectrum are acquired based on real speech data. The text feature module 52 is used to input the text sequence into a text encoder and output the text feature data of the text sequence. The spectrum feature module 53 is used to input the linear spectrum into a variational autoencoder and output the spectrum feature data of the linear spectrum. The alignment position calculation module 54 is used to... The system obtains alignment position calculation data for the training samples based on the text feature data and the spectral feature data; a first training module 55 is used to input the text feature data into the alignment position prediction model and output the alignment position prediction data corresponding to the text feature data; calculate the prediction error based on the alignment position calculation data and the alignment position prediction data, and adjust the parameters of the alignment position prediction model based on the prediction error until the alignment position prediction model reaches the training convergence condition; a second training module 56 is used to input the alignment position calculation data into the reconstruction mapping model and output the spectral feature prediction data corresponding to the alignment position calculation data; obtain the KL divergence between the spectral feature data and the spectral feature prediction data, and adjust the parameters of the reconstruction mapping model based on the KL divergence until the reconstruction mapping model reaches the training convergence condition.

[0160] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 6 As shown, the electronic device 60 includes a processor 61 and a memory 62 coupled to the processor 61.

[0161] The memory 62 stores program instructions for implementing the speech synthesis method or the training method of the speech synthesis model of any of the above embodiments.

[0162] The processor 61 is used to execute program instructions stored in the memory 62 for speech synthesis or training of a speech synthesis model.

[0163] The processor 61 can also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0164] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a storage medium according to an embodiment of this application. The storage medium 70 of this embodiment stores program instructions 71 capable of implementing all the above methods. These program instructions 71 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0166] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0167] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the text feature data of the text sequence to be predicted; The text feature data is input into a pre-trained alignment position prediction model, and the alignment position prediction data corresponding to the text feature data is output. The alignment position prediction model is trained based on real text sequences and real linear spectra obtained from real speech data. The alignment position prediction data is input into a pre-trained reconstruction mapping model, and the spectral feature prediction data corresponding to the alignment position prediction data is output. The reconstruction mapping model is trained based on real text sequences and real linear spectra obtained from real speech data. The training steps of the alignment position prediction model include: acquiring at least one training sample, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being acquired based on real speech data; inputting the text sequence into a text encoder and outputting text feature data of the text sequence; inputting the linear spectrum into a variational autoencoder and outputting spectral feature data of the linear spectrum; acquiring alignment position calculation data of the training sample based on the text feature data and the spectral feature data; inputting the text feature data into the alignment position prediction model and outputting alignment position prediction data corresponding to the text feature data; calculating the prediction error based on the alignment position calculation data and the alignment position prediction data; and adjusting the parameters of the alignment position prediction model based on the prediction error until the alignment position prediction model reaches the training convergence condition. The training steps of the reconstruction mapping model include: acquiring at least one training sample, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being acquired based on real speech data; inputting the text sequence into a text encoder and outputting text feature data of the text sequence; inputting the linear spectrum into a variational autoencoder and outputting spectral feature data of the linear spectrum, wherein the spectral feature data is the mean and variance of a multivariate Gaussian distribution; acquiring alignment position calculation data of the training sample based on the text feature data and the spectral feature data; inputting the alignment position calculation data into the reconstruction mapping model and outputting spectral feature prediction data corresponding to the alignment position calculation data; acquiring the KL divergence of the spectral feature data and the spectral feature prediction data, and adjusting the parameters of the reconstruction mapping model based on the KL divergence until the reconstruction mapping model reaches the training convergence condition.

2. The speech synthesis method according to claim 1, characterized in that, The step of obtaining the alignment position calculation data of the training samples based on the text feature data and the spectral feature data includes: The first alignment matrix of the text feature data is obtained based on the text feature data and the spectral feature data; The index mapping vector of the spectral feature data is obtained based on the first alignment matrix and the text feature data; The second alignment matrix of the spectral feature data is obtained based on the index mapping vector and the text feature data; The alignment position calculation data of the text feature data is obtained based on the second alignment matrix and the spectral feature data.

3. The speech synthesis method according to claim 2, characterized in that, After inputting the alignment position prediction data into the pre-trained reconstruction mapping model and outputting the spectral feature prediction data corresponding to the alignment position prediction data, the method further includes: The spectral feature prediction data is input into the speech generator, which outputs the corresponding speech synthesis data. The training steps for the speech generator include: Obtain at least one training sample, the training sample comprising real speech data and a linear spectrum extracted from the real speech data; The linear spectrum is input into a variational autoencoder, which outputs the spectral feature data of the linear spectrum. The spectral feature data is input into the speech generator, which outputs synthesized speech data. The synthesis error is calculated based on the real speech data and the synthesized speech data. The parameters of the speech generator are then adjusted based on the synthesis error until the speech generator reaches the training convergence condition.

4. A training method for a speech synthesis model, characterized in that, The speech synthesis model includes an alignment position prediction model and a reconstruction mapping model, and the training method includes: At least one training sample is obtained, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being obtained based on real speech data; The text sequence is input into a text encoder, which outputs the text feature data of the text sequence. The linear spectrum is input to a variational autoencoder, which outputs spectral feature data of the linear spectrum, wherein the spectral feature data is the mean and variance of a multivariate Gaussian distribution; The alignment position calculation data of the training samples is obtained based on the text feature data and the spectral feature data; The text feature data is input into the alignment position prediction model, and the alignment position prediction data corresponding to the text feature data is output. The prediction error is calculated based on the alignment position calculation data and the alignment position prediction data. The parameters of the alignment position prediction model are adjusted based on the prediction error until the alignment position prediction model reaches the training convergence condition. The alignment position calculation data is input into the reconstruction mapping model, and the spectral feature prediction data corresponding to the alignment position calculation data is output. Obtain the KL divergence of the spectral feature data and the spectral feature prediction data, and adjust the parameters of the reconstructed mapping model according to the KL divergence until the reconstructed mapping model reaches the training convergence condition.

5. The training method for the speech synthesis model according to claim 4, characterized in that, The speech synthesis model also includes a speech generator, and the training samples also include the real speech data; The training method also includes: The spectral feature data is input into the speech generator, which outputs synthesized speech data. The synthesis error is calculated based on the real speech data and the synthesized speech data. The parameters of the speech generator are then adjusted based on the synthesis error until the speech generator reaches the training convergence condition.

6. A speech synthesis device, characterized in that, include: The text feature extraction module is used to obtain the text feature data of the text sequence to be predicted; The alignment position prediction module is used to input the text feature data into a pre-trained alignment position prediction model and output the alignment position prediction data corresponding to the text feature data. The alignment position prediction model is trained based on real text sequences and real linear spectra obtained from real speech data. The mapping module is used to input the alignment position prediction data into a pre-trained reconstruction mapping model and output the spectral feature prediction data corresponding to the alignment position prediction data. The reconstruction mapping model is trained based on real text sequences and real linear spectra obtained from real speech data. The training steps of the alignment position prediction model include: acquiring at least one training sample, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being acquired based on real speech data; inputting the text sequence into a text encoder and outputting text feature data of the text sequence; inputting the linear spectrum into a variational autoencoder and outputting spectral feature data of the linear spectrum; acquiring alignment position calculation data of the training sample based on the text feature data and the spectral feature data; inputting the text feature data into the alignment position prediction model and outputting alignment position prediction data corresponding to the text feature data; calculating the prediction error based on the alignment position calculation data and the alignment position prediction data; and adjusting the parameters of the alignment position prediction model based on the prediction error until the alignment position prediction model reaches the training convergence condition. The training steps of the reconstruction mapping model include: acquiring at least one training sample, the training sample including a text sequence and a linear spectrum corresponding to the text sequence, the text sequence and the linear spectrum being acquired based on real speech data; inputting the text sequence into a text encoder and outputting text feature data of the text sequence; inputting the linear spectrum into a variational autoencoder and outputting spectral feature data of the linear spectrum, wherein the spectral feature data is the mean and variance of a multivariate Gaussian distribution; acquiring alignment position calculation data of the training sample based on the text feature data and the spectral feature data; inputting the alignment position calculation data into the reconstruction mapping model and outputting spectral feature prediction data corresponding to the alignment position calculation data; acquiring the KL divergence of the spectral feature data and the spectral feature prediction data, and adjusting the parameters of the reconstruction mapping model based on the KL divergence until the reconstruction mapping model reaches the training convergence condition.

7. An electronic device, characterized in that, The system includes a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; when the processor executes the program instructions stored in the memory, it implements the speech synthesis method as described in any one of claims 1 to 3 or the training method for the speech synthesis model as described in claim 4 or 5.

8. A storage medium, characterized in that, The storage medium stores program instructions, which, when executed by a processor, implement the speech synthesis method as described in any one of claims 1 to 3 or the training method for the speech synthesis model as described in claim 4 or 5.

Images