Speech synthesis method, speech synthesis model training method, speech synthesis model training device and storage medium

Abstract

The invention provides a speech synthesis model training method, a speech synthesis method, a speech synthesis device and a computer readable storage medium. The speech synthesis model training method comprises the following steps: inputting a speech text into a speech synthesis model to be trained, and obtaining predicted acoustic features; obtaining a spectrum similarity loss value based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix; obtaining an energy distribution loss value based on an actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix; and training the speech synthesis model according to the spectrum similarity loss value and the energy distribution loss value. Through the above mode, the speech synthesis device trains the speech synthesis model through the frequency spectrum similarity loss function and the energy distribution loss function, so as to alleviate the problems of overall plain and local flaws of the synthesized speech, and improve the speech synthesis effect.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis model training method, a speech synthesis method, a speech synthesis device, and a computer-readable storage medium. Background Art

[0002] Text To Speech (TTS) is a technology that can convert spoken text into speech and audio signals. TTS systems generally include three functional modules: linguistic feature extraction module, acoustic model, and vocoder. This solution is mainly aimed at optimizing and improving the acoustic model. The function of the acoustic model is to convert linguistic features (such as phoneme sequences) into acoustic features (such as Mel spectrum). The existing mainstream algorithm solutions include Tacotron, TransformerTTS, FastSpeech, DelightfulTTS, etc., as well as their improved solutions.

[0003] The acoustic model plays a key role in the TTS system. It not only solves the problem of speech signal generation, but also determines the voice timbre, controls rhythm and emotion. However, when training this key model, the loss function used by mainstream algorithms is often very simple. For example, the mean absolute error (MAE) is directly used as the spectrum loss function, that is, the MAE value of the Mel spectrum of the real speech and the Mel spectrum predicted by the model is calculated. This loss function pays more attention to the global characteristics of the audio, and the detailed characteristics of the audio are easily covered up, causing the overall voice expression to be bland or even local pronunciation defects, which greatly affects the listening experience. Summary of the invention

[0004] The present application provides a speech synthesis model training method, a speech synthesis method, a speech synthesis device and a computer-readable storage medium.

[0005] The present application provides a model training method for speech synthesis, and the model training method for speech synthesis comprises:

[0006] Input the speech text into the speech synthesis model to be trained to obtain the predicted acoustic features;

[0007] Acquiring actual acoustic features corresponding to the speech text;

[0008] Acquire an actual spectral energy high-density subspace matrix and an actual spectral energy low-density subspace matrix of the actual acoustic feature, and a predicted spectral energy high-density subspace matrix and a predicted spectral energy low-density subspace matrix of the predicted acoustic feature;

[0009] Acquire a spectrum similarity loss value based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix;

[0010] Acquire an energy distribution loss value based on the actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix;

[0011] The speech synthesis model is trained according to the spectrum similarity loss value and the energy distribution loss value.

[0012] The step of obtaining the actual spectrum energy high-density subspace matrix and the actual spectrum energy low-density subspace matrix of the actual acoustic feature includes:

[0013] Acquiring a frequency spectrum matrix of the actual acoustic feature;

[0014] Calculating the average value of all elements in the spectrum matrix;

[0015] setting a threshold value according to the average value;

[0016] Using the elements greater than or equal to the threshold among all the elements, constructing the actual spectrum energy high-density subspace matrix;

[0017] The actual spectrum energy low-density subspace matrix is ​​constructed by using the elements smaller than the threshold among all the elements.

[0018] Wherein, after obtaining the spectrum matrix of the actual acoustic features, the model training method further includes:

[0019] Obtaining the minimum element and the maximum element in the spectrum matrix;

[0020] All elements of the frequency spectrum matrix are normalized by using the minimum element and the maximum element to obtain a normalized frequency spectrum matrix.

[0021] The acquiring of the spectrum similarity loss value based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix includes:

[0022] Obtaining an actual high-density vector of the actual spectrum energy high-density subspace matrix;

[0023] Obtaining a predicted high-density vector of the predicted spectral energy high-density subspace matrix;

[0024] The spectrum similarity loss value is obtained by using the covariance and standard deviation of the actual high-density vector and the predicted high-density vector.

[0025] The columns of the actual spectrum energy high-density subspace matrix are divided according to the spectrum length, and the rows are divided according to the spectrum feature dimension;

[0026] The obtaining of the actual high-density vector of the actual spectrum energy high-density subspace matrix comprises:

[0027] The actual spectrum energy high-density subspace matrix is ​​expanded in the direction of rows and spliced ​​into a one-dimensional actual high-density vector.

[0028] The step of acquiring the energy distribution loss value based on the actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix includes:

[0029] Dividing the actual spectrum energy low-density subspace matrix into a plurality of actual low-density sub-matrices, and dividing the predicted spectrum energy low-density subspace matrix into a plurality of predicted low-density sub-matrices of the same number;

[0030] Obtain the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position;

[0031] The energy distribution loss value is taken as the average value of all relative entropies.

[0032] The step of obtaining the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position includes:

[0033] Obtaining an actual mean and an actual variance of all elements in the actual low-density submatrix, and obtaining an actual probability density function of the actual low-density submatrix according to the actual mean and the actual variance;

[0034] Obtaining predicted means and predicted variances of all elements in the predicted low-density submatrix, and obtaining a predicted probability density function of the predicted low-density submatrix based on the predicted means and the predicted variance;

[0035] The relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position is determined by using the actual probability density function and the predicted probability density function.

[0036] The step of obtaining the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position includes:

[0037] Obtaining a preset sampling rate, and determining the number of samples according to the preset sampling rate and the number of actual low-density sub-matrices;

[0038] Randomly acquiring a random actual low-density sub-matrix from the plurality of actual low-density sub-matrices according to the sampling quantity, and a random predicted low-density sub-matrix at a corresponding position;

[0039] Get the relative entropy of a random actual low-density submatrix and a random predicted low-density submatrix at the same position.

[0040] The present application also provides a speech synthesis method, the speech synthesis method comprising:

[0041] Get the speech text to be synthesized;

[0042] Input the speech text to be synthesized into a pre-trained speech synthesis model to obtain the speech audio corresponding to the speech text;

[0043] Wherein, the speech synthesis model is trained by the above-mentioned model training method.

[0044] The present application also provides a speech synthesis device, which includes a processor and a memory, wherein the memory stores program data, and the processor is used to execute the program data to implement the model training method for speech synthesis and / or the speech synthesis method as described above.

[0045] The present application also provides a computer-readable storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the above-mentioned model training method for speech synthesis and / or speech synthesis method.

[0046] The beneficial effects of the present application are as follows: the speech synthesis device inputs the speech text into the speech synthesis model to be trained to obtain the predicted acoustic features; obtains the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features, as well as the predicted spectral energy high-density subspace matrix and the predicted spectral energy low-density subspace matrix of the predicted acoustic features; obtains the spectral similarity loss value based on the actual spectral energy high-density subspace matrix and the predicted spectral energy high-density subspace matrix; obtains the energy distribution loss value based on the actual spectral energy low-density subspace matrix and the predicted spectral energy low-density subspace matrix; trains the speech synthesis model according to the spectral similarity loss value and the energy distribution loss value. In the above manner, the speech synthesis device trains the speech synthesis model through the spectral similarity loss function and the energy distribution loss function to alleviate the problems of overall blandness and local defects of the synthesized speech and improve the speech synthesis effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0048] Figure 1 It is a flowchart of an embodiment of a model training method for speech synthesis provided by the present application;

[0049] Figure 2 is a schematic diagram of the audio spectrum diagram and its splitting results provided by this application;

[0050] Figure 3 yes Figure 1 A schematic diagram of a specific flow chart of step S13 of the model training method shown;

[0051] Figure 4 It is a flowchart of an embodiment of a speech synthesis method provided by the present application;

[0052] Figure 5 It is a structural schematic diagram of an embodiment of a speech synthesis device provided by the present application;

[0053] Figure 6 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Different from the existing method of directly using MAE to evaluate the distance between the real speech Mel spectrum and the model predicted Mel spectrum, this application proposes, based on MAE, to calculate the spectrum similarity loss in the high-density subspace of spectrum energy and the energy distribution loss in the low-density subspace of spectrum energy to alleviate the problem of overall blandness and local defects in synthesized speech.

[0056] Please refer to Figure 1 , Figure 1 It is a flow chart of an embodiment of a model training method for speech synthesis provided in the present application.

[0057] The model training method for speech synthesis of the present application is applied to a speech synthesis device, wherein the speech synthesis device of the present application can be a server, or a system composed of a server and a terminal device. Accordingly, the various parts of the speech synthesis device, such as various units, subunits, modules, and submodules, can all be set in the server, or can be set in the server and the terminal device respectively.

[0058] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the model training method for speech synthesis of the embodiment of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0059] Specifically, if Figure 1 As shown, the model training method for speech synthesis in the embodiment of the present application specifically includes the following steps:

[0060] Step S11: input the speech text into the speech synthesis model to be trained to obtain predicted acoustic features.

[0061] Step S12: Obtain the actual acoustic features corresponding to the speech text.

[0062] In the embodiment of the present application, both the actual acoustic features and the predicted acoustic features are embodied by audio spectrograms or spectrum matrices, wherein the actual acoustic features are the real audio spectrograms corresponding to the speech text, and the predicted acoustic features are the predicted audio spectrograms output by the speech synthesis model according to the speech text.

[0063] Step S13: obtaining an actual spectral energy high-density subspace matrix and an actual spectral energy low-density subspace matrix of actual acoustic features, and a predicted spectral energy high-density subspace matrix and a predicted spectral energy low-density subspace matrix of predicted acoustic features.

[0064] In an embodiment of the present application, in order to calculate the subsequent spectral similarity loss and energy distribution loss, the speech synthesis device needs to split the audio spectrum graph into two parts: a high-light matrix representing a high-density subspace of spectral energy, and a low-light matrix representing a low-density subspace of spectral energy.

[0065] Specifically, the actual audio spectrogram of the actual acoustic feature can be split into an actual spectral energy high-density subspace matrix and an actual spectral energy low-density subspace matrix, and the predicted audio spectrogram of the predicted acoustic feature can be split into a predicted spectral energy high-density subspace matrix and a predicted spectral energy low-density subspace matrix.

[0066] The following is an example of the splitting process of the actual audio spectrogram of the actual acoustic feature. It should be noted that the splitting process of the predicted audio spectrogram of the predicted acoustic feature is consistent with the splitting process of the actual audio spectrogram of the actual acoustic feature.

[0067] Please refer to Figure 2 , Figure 2 is a schematic diagram of the audio spectrum diagram and its splitting results provided by this application. Figure 2 As shown, (A) shows the original audio spectrum; (B) is the high-density subspace of the spectrum energy, from which the highlighted harmonic structure can be clearly seen; (C) is the low-density subspace of the spectrum energy, from which the distribution of energy in each area can be clearly seen.

[0068] Combination Figure 2 Please continue to refer to the split results Figure 3 , Figure 3 yes Figure 1 A specific flow chart of step S13 of the model training method is shown.

[0069] Specifically, if Figure 3 As shown, the model training method for speech synthesis in the embodiment of the present application specifically includes the following steps:

[0070] Step S131: Calculate the average value of all elements in the spectrum matrix.

[0071] In the embodiment of the present application, the speech synthesis device obtains the frequency spectrum matrix of the actual acoustic features and calculates the average value of all elements.

[0072] Furthermore, before processing the spectrum matrix of the actual acoustic features, the speech synthesis device may also normalize or regularize the spectrum matrix in order to improve the matrix processing speed and accuracy.

[0073] Specifically, the speech synthesis device normalizes the elements in the spectrum matrix to the range [0, 1]. There are many normalization methods, and a simple linear normalization method is used as an example. Let the smallest element of the matrix be a min , the largest element is a max , then the normalization function is: f norm (a i )=(a i -a mix ) / (a max -a min ), the normalized matrix is ​​denoted as A normed .

[0074] In other implementations, the speech synthesis device may also adopt other nonlinear regularization methods, such as a sigmoid function, etc., and all feasible solutions are not listed here.

[0075] Step S132: setting a threshold according to the average value.

[0076] In the embodiment of the present application, the speech synthesis device sets the threshold according to the average value. For example, the threshold THR that can be selected is 1.2 times the average value, that is, THR=1.2×a avg Among them, a avg is the matrix A normed The average value of all elements in .

[0077] It should be noted that when splitting the spectrum matrix into a high-light matrix and a low-light matrix, it is not necessary to use the above-mentioned solution of setting the threshold based on experience. It can also be done by filtering and other methods as long as the harmonic structure can be clearly and completely separated.

[0078] Step S133: constructing a high-density subspace matrix of actual spectrum energy using elements greater than or equal to the threshold among all elements.

[0079] In the embodiment of the present application, the speech synthesis device splits the high-light matrix, that is, the matrix A normed All elements in A that are less than the threshold THR can be set to 0. normed Generate high-light matrix A high The function can be expressed as:

[0080]

[0081] Step S134: constructing a low-density subspace matrix of actual spectrum energy using elements smaller than a threshold value among all elements.

[0082] In the embodiment of the present application, the speech synthesis device splits the low-light matrix, that is, the matrix A normed All elements greater than or equal to the threshold THR can be set to 0. normed Generate low-light matrix A low The function can be expressed as:

[0083]

[0084] Step S14: obtaining a spectrum similarity loss value based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix.

[0085] In the embodiment of the present application, since there is an obvious correlation between adjacent frames and harmonics in the spectrum diagram, the correlation coefficient of the spectrum matrix can be used to represent the similarity. The similarity loss L similar It can be calculated as follows:

[0086]

[0087] Among them, a and b represent A respectively.high ,B high The vector representation of ; Cov(a,b) is the covariance of a and b; σ a ,σ b is the standard deviation of a and b; C is a very small positive real number. high is the actual spectrum energy high-density subspace matrix, B high A high-density subspace matrix for predicting spectral energy.

[0088] Specifically, the present application also provides a matrix vectorization representation method, that is, expanding the matrix row by row and concatenating them into vectors, which is specifically represented as follows:

[0089]

[0090] a=[a 11 ,a 12 ,...,a 1T ,a 21 ,a 22 ,...,a 2T ,...,a D1 ,a D2 ,...,a DT ]

[0091] Where D is the spectral feature dimension, and T is the spectrum length (i.e., the number of speech frames).

[0092] Why must the spectral similarity loss be calculated on the high-light matrix instead of directly on the original spectrum matrix? Because the main components of speech content and timbre are expressed as fundamental waves, harmonics, and the relative distance between harmonics on the spectrum graph, the goal of spectral similarity loss is to quantify the difference between these main elements on the two spectrum graphs. When the audio is noisy, these main elements will be mixed with the noise energy on the spectrum graph and it is difficult to distinguish, especially in the early stage of model training, when the spectrum generated by the model is very fuzzy. Therefore, when calculating the spectral similarity loss, it is necessary to calculate it on the high-light matrix.

[0093] Step S15: acquiring an energy distribution loss value based on the actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix.

[0094] In the embodiment of the present application, if all elements of the spectrum matrix, that is, matrix A normedIf the energy distribution is calculated on the high-light matrix, the result will be significantly affected by the highlighted harmonic structure, that is, the elements corresponding to the high-light matrix. In order to enable the speech synthesis model to generate speech quality details more accurately, the energy distribution loss is only calculated on the low-light matrix. Specifically, the algorithm idea of ​​the energy distribution loss function provided in this application is to calculate the KL divergence of the spectrum matrix in blocks, that is, the relative entropy and then take the average value.

[0095] Specifically, the calculation process of the energy distribution loss value is as follows:

[0096] (1) Split the sub-matrices. The low-light matrix obtained from the Mel spectrum matrix of the real speech is denoted as A low , the low-light matrix obtained by the Mel spectrum matrix predicted by the model is recorded as B low . low , B low Divide into multiple sub-matrices of size M×N, and fill the insufficient edges with 0. The sub-matrices are recorded as and i∈[1,L]. M×N can be set empirically, such as 10×8, etc., and L is the number of sub-matrices.

[0097] (2) Calculate the distribution function of the submatrix. Assuming that the elements in each submatrix obey a Gaussian distribution, its mean and variance can be estimated to obtain its probability density function. The probability density function of Submatrix The probability density function of

[0098] (3) Calculate the energy distribution loss. Calculate the KL divergence of each pair of sub-matrices with the same position and then take the average value, which is used as the energy distribution loss. That is:

[0099]

[0100] Where D KL is the KL divergence, and L is the number of submatrices.

[0101] In other implementations, since calculating the KL divergence of all submatrix pairs will bring more computational complexity, in actual model training, random sampling of submatrix pairs can be used instead of full calculation. When there are enough data samples for training and enough training iterations, the effects of random sampling and full calculation are equivalent.

[0102] Specifically, the speech synthesis device first sets a sampling rate r, such as r = 0.1. According to the sampling rate, the sampling sub-matrix logarithm L' = rL is calculated. For each pair of sampled sub-matrices in A low , Blow The sampling position is random, as long as the submatrix does not exceed the boundary, that is, the coordinates of the starting position (x, y) = (rand (1, DM), rand (1, TN)), where rand (a, b) means randomly selecting an integer in the integer range [a, b].

[0103] Step S16: training the speech synthesis model according to the spectrum similarity loss value and the energy distribution loss value.

[0104] In the embodiment of the present application, after obtaining the spectrum similarity loss L similar and energy distribution loss L distrib Finally, the spectral loss function for training the speech synthesis model proposed in this application can be specifically expressed as:

[0105] L=w1·MAE(A,B)+w2·L similar (A high ,B high )+w3·L distrib (A low ,B low )

[0106] Among them, w1, w2, w3 are the weights of each sub-item, which default to 1 and can be adjusted based on experience.

[0107] The speech synthesis device inputs the speech text into the speech synthesis model to be trained to obtain the predicted acoustic features; obtains the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features, as well as the predicted spectral energy high-density subspace matrix and the predicted spectral energy low-density subspace matrix of the predicted acoustic features; obtains the spectrum similarity loss value based on the actual spectral energy high-density subspace matrix and the predicted spectral energy high-density subspace matrix; obtains the energy distribution loss value based on the actual spectral energy low-density subspace matrix and the predicted spectral energy low-density subspace matrix; trains the speech synthesis model according to the spectrum similarity loss value and the energy distribution loss value. In the above manner, the speech synthesis device trains the speech synthesis model through the spectral similarity loss function and the energy distribution loss function to alleviate the problems of overall blandness and local defects of the synthesized speech and improve the speech synthesis effect.

[0108] This application proposes a spectrum similarity loss to evaluate the difference in harmonic structure between the model-generated spectrum and the real speech spectrum; proposes an energy distribution loss to evaluate the difference in timbre details between the model-generated spectrum and the real speech spectrum; and proposes an improved scheme for random sampling submatrices, which greatly reduces the amount of calculation without affecting the model training results.

[0109] based on Figure 1Please continue to refer to the model training method for speech synthesis shown in Figure 4 , Figure 4 It is a flowchart of an embodiment of a speech synthesis method provided by the present application.

[0110] Specifically, if Figure 4 As shown, the model training method for speech synthesis in the embodiment of the present application specifically includes the following steps:

[0111] Step S21: Obtain the speech text to be synthesized.

[0112] Step S22: input the speech text to be synthesized into a pre-trained speech synthesis model to obtain the speech audio corresponding to the speech text.

[0113] In the embodiment of the present application, the pre-trained speech synthesis model is Figure 1 The model training method shown is obtained by training, and the functions of the relevant speech synthesis model have been reflected in the training process, which will not be repeated here.

[0114] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0115] In order to implement the model training method and speech synthesis method of the above embodiment, the present application also proposes a speech synthesis device, which can be found in detail in Figure 5 , Figure 5 It is a structural diagram of an embodiment of a speech synthesis device provided in the present application.

[0116] The speech synthesis device 300 of the embodiment of the present application includes a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled.

[0117] The memory 31 is used to store program data, and the processor 32 is used to execute the program data to implement the model training method for speech synthesis and the speech synthesis method described in the above embodiments.

[0118] In this embodiment, the processor 32 may also be referred to as a CPU (Central Processing Unit). The processor 32 may be an integrated circuit chip having signal processing capabilities. The processor 32 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 32 may also be any conventional processor, etc.

[0119] In order to implement the model training method and speech synthesis method of the above-mentioned embodiment, the present application also provides a computer-readable storage medium, such as Figure 6 As shown, the computer-readable storage medium 400 is used to store program data 41. When the program data 41 is executed by the processor, it is used to implement the model training method and speech synthesis method of the speech synthesis as described in the above embodiments.

[0120] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the model training method for speech synthesis and the speech synthesis method as described in the embodiments of the present application. The computer program product can be a software installation package.

[0121] The model training method and speech synthesis method of the speech synthesis described in the above embodiments of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0122] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A model training method for speech synthesis, characterized in that: The model training method for speech synthesis includes: Input the speech text into the speech synthesis model to be trained to obtain the predicted acoustic features; Acquiring actual acoustic features corresponding to the speech text; Acquire an actual spectral energy high-density subspace matrix and an actual spectral energy low-density subspace matrix of the actual acoustic feature, and a predicted spectral energy high-density subspace matrix and a predicted spectral energy low-density subspace matrix of the predicted acoustic feature; Acquire a spectrum similarity loss value based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix; Acquire an energy distribution loss value based on the actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix; The speech synthesis model is trained according to the spectrum similarity loss value and the energy distribution loss value.

2. The model training method according to claim 1, characterized in that: The step of obtaining the actual spectrum energy high-density subspace matrix and the actual spectrum energy low-density subspace matrix of the actual acoustic feature includes: Acquiring a frequency spectrum matrix of the actual acoustic feature; Calculating the average value of all elements in the spectrum matrix; setting a threshold value according to the average value; Using the elements greater than or equal to the threshold among all the elements, constructing the actual spectrum energy high-density subspace matrix; The actual spectrum energy low-density subspace matrix is ​​constructed by using the elements smaller than the threshold among all the elements.

3. The model training method according to claim 2, characterized in that: After obtaining the spectrum matrix of the actual acoustic features, the model training method further includes: Obtaining the minimum element and the maximum element in the spectrum matrix; All elements of the frequency spectrum matrix are normalized by using the minimum element and the maximum element to obtain a normalized frequency spectrum matrix.

4. The model training method according to claim 1, characterized in that: The acquiring of the spectrum similarity loss value based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix comprises: Obtaining an actual high-density vector of the actual spectrum energy high-density subspace matrix; Obtaining a predicted high-density vector of the predicted spectral energy high-density subspace matrix; The spectrum similarity loss value is obtained by using the covariance and standard deviation of the actual high-density vector and the predicted high-density vector.

5. The model training method according to claim 4, characterized in that: The columns of the actual spectrum energy high-density subspace matrix are divided according to the spectrum length, and the rows are divided according to the spectrum feature dimension; The obtaining of the actual high-density vector of the actual spectrum energy high-density subspace matrix comprises: The actual spectrum energy high-density subspace matrix is ​​expanded in the direction of rows and spliced ​​into a one-dimensional actual high-density vector.

6. The model training method according to claim 1, characterized in that: The acquiring the energy distribution loss value based on the actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix includes: Dividing the actual spectrum energy low-density subspace matrix into a plurality of actual low-density sub-matrices, and dividing the predicted spectrum energy low-density subspace matrix into a plurality of predicted low-density sub-matrices of the same number; Obtain the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position; The energy distribution loss value is taken as the average value of all relative entropies.

7. The model training method according to claim 6, characterized in that: The step of obtaining the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position includes: Obtaining an actual mean and an actual variance of all elements in the actual low-density submatrix, and obtaining an actual probability density function of the actual low-density submatrix according to the actual mean and the actual variance; Obtaining predicted means and predicted variances of all elements in the predicted low-density submatrix, and obtaining a predicted probability density function of the predicted low-density submatrix based on the predicted means and the predicted variance; The relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position is determined by using the actual probability density function and the predicted probability density function.

8. The model training method according to claim 6, characterized in that: The step of obtaining the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position includes: Obtaining a preset sampling rate, and determining the number of samples according to the preset sampling rate and the number of actual low-density sub-matrices; Randomly acquiring a random actual low-density sub-matrix from the plurality of actual low-density sub-matrices according to the sampling quantity, and a random predicted low-density sub-matrix at a corresponding position; Get the relative entropy of a random actual low-density submatrix and a random predicted low-density submatrix at the same position.

9. A speech synthesis method, characterized in that: The speech synthesis method comprises: Get the speech text to be synthesized; Input the speech text to be synthesized into a pre-trained speech synthesis model to obtain the speech audio corresponding to the speech text; Wherein, the speech synthesis model is trained by the model training method described in any one of claims 1 to 8.

10. A speech synthesis device, characterized in that: The speech synthesis device includes a processor and a memory, wherein the memory stores program data, and the processor is used to execute the program data to implement the model training method according to any one of claims 1 to 8, and / or the speech synthesis method according to claim 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program data, and when the program data is executed by the processor, it is used to implement the model training method described in any one of claims 1-8 and / or the speech synthesis method described in claim 9.

Images