Speech synthesis methods, model training methods, devices, and storage media

Abstract

This application provides a method for training a speech synthesis model, a speech synthesis method, a speech synthesis device, and a computer-readable storage medium. The speech synthesis model training method includes: inputting spoken text into a speech synthesis model to be trained to obtain predicted acoustic features; obtaining a spectral similarity loss value based on the actual spectral energy high-density subspace matrix and the predicted spectral energy high-density subspace matrix; obtaining an energy distribution loss value based on the actual spectral energy low-density subspace matrix and the predicted spectral energy low-density subspace matrix; and training the speech synthesis model according to the spectral similarity loss value and the energy distribution loss value. Through this method, the speech synthesis device trains the speech synthesis model using a spectral similarity loss function and an energy distribution loss function to alleviate the problems of overall blandness and local flaws in synthesized speech, thereby improving the speech synthesis effect.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech synthesis model training method, a speech synthesis method, a speech synthesis device, and a computer-readable storage medium. Background Technology

[0002] Text-to-Speech (TTS) is a technology that converts spoken text into audio signals. A typical TTS system comprises three functional modules: a linguistic feature extraction module, an acoustic model, and a vocoder. This solution primarily focuses on optimizing and improving the acoustic model, which converts linguistic features (such as phoneme sequences) into acoustic features (such as Mel-spectrum data). Existing mainstream algorithms include Tacotron, TransformerTTS, FastSpeech, DelightfulTTS, and their improved versions.

[0003] Acoustic models play a crucial role in TTS systems, not only in generating speech signals but also in determining timbre, controlling prosody, and conveying emotion. However, mainstream algorithms often use very simple loss functions when training this critical model. For example, they directly use Mean Absolute Error (MAE) as the spectral loss function, calculating the MAE value between the Mel spectrum of the real speech and the Mel spectrum predicted by the model. This type of loss function focuses more on global audio characteristics, easily masking detailed audio features, resulting in a flat overall speech performance or even localized pronunciation flaws, significantly impacting the listening experience. Summary of the Invention

[0004] This application provides a method for training a speech synthesis model, a speech synthesis method, a speech synthesis device, and a computer-readable storage medium.

[0005] This application provides a method for training a speech synthesis model, the method comprising:

[0006] Input the spoken text into the speech synthesis model to be trained to obtain the predicted acoustic features;

[0007] Obtain the actual acoustic features corresponding to the spoken text;

[0008] Obtain the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features, as well as the predicted spectral energy high-density subspace matrix and the predicted spectral energy low-density subspace matrix of the predicted acoustic features;

[0009] Based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix, obtain the spectrum similarity loss value;

[0010] Based on the actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix, the energy distribution loss value is obtained;

[0011] The speech synthesis model is trained according to the spectral similarity loss value and the energy distribution loss value.

[0012] The step of obtaining the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features includes:

[0013] Obtain the spectral matrix of the actual acoustic features;

[0014] Calculate the average value of all elements in the spectrum matrix;

[0015] Set the threshold according to the average value;

[0016] The actual spectral energy high-density subspace matrix is ​​constructed using all elements that are greater than or equal to the threshold.

[0017] The actual spectral energy low-density subspace matrix is ​​constructed using all elements that are less than the threshold.

[0018] After obtaining the spectral matrix of the actual acoustic features, the model training method further includes:

[0019] Obtain the minimum and maximum elements in the spectrum matrix;

[0020] The minimum and maximum elements are used to normalize all elements of the spectrum matrix to obtain the normalized spectrum matrix.

[0021] The step of obtaining the spectral similarity loss value based on the actual spectral energy high-density subspace matrix and the predicted spectral energy high-density subspace matrix includes:

[0022] Obtain the actual high-density vector of the actual spectral energy high-density subspace matrix;

[0023] Obtain the predicted high-density vector of the predicted spectral energy high-density subspace matrix;

[0024] The spectral similarity loss value is obtained by using the covariance and standard deviation of the actual high-density vector and the predicted high-density vector.

[0025] The columns of the actual spectral energy high-density subspace matrix are divided according to the spectral length, and the rows are divided according to the spectral feature dimension.

[0026] The step of obtaining the actual high-density vector of the actual spectral energy high-density subspace matrix includes:

[0027] The actual spectral energy high-density subspace matrix is ​​expanded along the row direction and spliced ​​into a one-dimensional actual high-density vector.

[0028] The step of obtaining the energy distribution loss value based on the actual spectral energy low-density subspace matrix and the predicted spectral energy low-density subspace matrix includes:

[0029] The actual spectral energy low-density subspace matrix is ​​divided into several actual low-density submatrices, and the predicted spectral energy low-density subspace matrix is ​​divided into the same number of predicted low-density submatrices.

[0030] Obtain the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position;

[0031] The energy distribution loss value is calculated as the average of all relative entropies.

[0032] The step of obtaining the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position includes:

[0033] Obtain the actual mean and actual variance of all elements in the actual low-density submatrix, and obtain the actual probability density function of the actual low-density submatrix based on the actual mean and actual variance.

[0034] Obtain the predicted mean and predicted variance of all elements in the predicted low-density submatrix, and obtain the predicted probability density function of the predicted low-density submatrix based on the predicted mean and predicted variance.

[0035] Using the actual probability density function and the predicted probability density function, the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position is determined.

[0036] The step of obtaining the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position includes:

[0037] Obtain a preset sampling rate, and determine the number of samples based on the preset sampling rate and the actual number of low-density submatrices;

[0038] According to the sampling quantity, a random actual low-density submatrix and a random predicted low-density submatrix at the corresponding position are randomly obtained from the plurality of actual low-density submatrices.

[0039] Obtain the relative entropy of random actual low-density submatrices and random predicted low-density submatrices at the same position.

[0040] This application also provides a speech synthesis method, the speech synthesis method comprising:

[0041] Obtain the text to be synthesized speech;

[0042] The speech text to be synthesized is input into a pre-trained speech synthesis model to obtain the speech audio corresponding to the speech text;

[0043] The speech synthesis model is trained using the model training method described above.

[0044] This application also provides a speech synthesis apparatus, which includes a processor and a memory. The memory stores program data, and the processor executes the program data to implement the speech synthesis model training method and / or speech synthesis method as described above.

[0045] This application also provides a computer-readable storage medium for storing program data, which, when executed by a processor, is used to implement the above-described speech synthesis model training method and / or speech synthesis method.

[0046] The beneficial effects of this application are as follows: The speech synthesis device inputs speech text into the speech synthesis model to be trained, and obtains predicted acoustic features; it obtains the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features, as well as the predicted spectral energy high-density subspace matrix and the predicted spectral energy low-density subspace matrix of the predicted acoustic features; based on the actual spectral energy high-density subspace matrix and the predicted spectral energy high-density subspace matrix, it obtains a spectral similarity loss value; based on the actual spectral energy low-density subspace matrix and the predicted spectral energy low-density subspace matrix, it obtains an energy distribution loss value; and it trains the speech synthesis model according to the spectral similarity loss value and the energy distribution loss value. Through the above method, the speech synthesis device trains the speech synthesis model using the spectral similarity loss function and the energy distribution loss function to alleviate the problems of overall blandness and local defects in synthesized speech, thereby improving the speech synthesis effect. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0048] Figure 1 This is a flowchart illustrating an embodiment of the speech synthesis model training method provided in this application;

[0049] Figure 2 This is a schematic diagram of the audio spectrogram and its decomposition results provided in this application;

[0050] Figure 3 yes Figure 1 A schematic diagram illustrating the specific process of step S13 in the model training method shown;

[0051] Figure 4 This is a flowchart illustrating an embodiment of the speech synthesis method provided in this application;

[0052] Figure 5 This is a schematic diagram of the structure of an embodiment of the speech synthesis device provided in this application;

[0053] Figure 6 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] Unlike existing methods that directly use MAE to evaluate the distance between the real speech Mel spectrum and the model-predicted Mel spectrum, this application proposes to calculate the spectral similarity loss in the high-density subspace of spectral energy and the energy distribution loss in the low-density subspace of spectral energy, in order to alleviate the problems of overall flatness and local defects in synthesized speech.

[0056] Please refer to the details. Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the speech synthesis model training method provided in this application.

[0057] The speech synthesis model training method of this application is applied to a speech synthesis device, which can be a server or a system consisting of a server and a terminal device working together. Accordingly, the various parts of the speech synthesis device, such as units, subunits, modules, and submodules, can all be located in the server, or they can be located separately in the server and the terminal device.

[0058] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed servers, or as a single software program or software module; no specific limitation is made here. In some possible implementations, the speech synthesis model training method of this application embodiment can be implemented by a processor calling computer-readable instructions stored in memory.

[0059] Specifically, such as Figure 1 As shown, the speech synthesis model training method of this application embodiment specifically includes the following steps:

[0060] Step S11: Input the speech text into the speech synthesis model to be trained and obtain the predicted acoustic features.

[0061] Step S12: Obtain the actual acoustic features corresponding to the speech text.

[0062] In this embodiment, both actual acoustic features and predicted acoustic features are represented by audio spectrograms or spectrum matrices. The actual acoustic features are the true audio spectrograms of the corresponding speech text, while the predicted acoustic features are the predicted audio spectrograms output by the speech synthesis model based on the speech text.

[0063] Step S13: Obtain the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features, as well as the predicted spectral energy high-density subspace matrix and the predicted spectral energy low-density subspace matrix of the predicted acoustic features.

[0064] In this embodiment of the application, in order to calculate the subsequent spectral similarity loss and energy distribution loss, the speech synthesis device needs to split the audio spectrogram into two parts: a high-light matrix representing the high-density subspace of spectral energy and a low-light matrix representing the low-density subspace of spectral energy.

[0065] Specifically, the actual audio spectrogram of the actual acoustic features can be decomposed into a high-density subspace matrix of actual spectral energy and a low-density subspace matrix of actual spectral energy, and the predicted audio spectrogram of the predicted acoustic features can be decomposed into a high-density subspace matrix of predicted spectral energy and a low-density subspace matrix of predicted spectral energy.

[0066] The following section will continue the explanation using the decomposition process of the actual audio spectrogram of actual acoustic features as an example. It should be noted that the decomposition process of the predicted audio spectrogram of predicted acoustic features is the same as the decomposition process of the actual audio spectrogram of actual acoustic features.

[0067] Please refer to the details. Figure 2 , Figure 2 This is a schematic diagram of the audio spectrogram and its decomposition results provided in this application. For example... Figure 2 As shown, (A) shows the original audio spectrum; (B) is the high-density subspace of the spectrum energy, from which the bright harmonic structure can be clearly seen; (C) is the low-density subspace of the spectrum energy, from which the distribution of energy in each region can be clearly seen.

[0068] Combination Figure 2 For the splitting results, please refer to [link / reference]. Figure 3 , Figure 3 yes Figure 1 The diagram shows the specific process flow of step S13 in the model training method shown.

[0069] Specifically, such as Figure 3 As shown, the speech synthesis model training method of this application embodiment specifically includes the following steps:

[0070] Step S131: Calculate the average value of all elements in the spectrum matrix.

[0071] In this embodiment of the application, the speech synthesis device acquires the spectrum matrix of the actual acoustic features and calculates the average value of all elements.

[0072] Furthermore, before processing the spectrum matrix of actual acoustic features, the speech synthesis device can normalize or regularize the spectrum matrix to improve the speed and accuracy of matrix processing.

[0073] Specifically, the speech synthesis device normalizes the elements in the spectral matrix to the range [0, 1]. There are various normalization methods; here, a simple linear normalization method is used as an example. Let the smallest element of matrix A be a. min The largest element is a max Then the normalization function is: f norm (a i )=(a i -a mix ) / (a max -a min Let A be the normalized matrix. normed .

[0074] In other implementations, the speech synthesis device may also employ other nonlinear normalization methods, such as the sigmoid function, etc., and feasible solutions will not be listed here.

[0075] Step S132: Set the threshold according to the average value.

[0076] In this embodiment, the speech synthesis device sets a threshold based on the average value. For example, the selectable threshold THR can be 1.2 times the average value, i.e., THR = 1.2 × a. avg Among them, a avg Let matrix A normed The average value of all elements in the matrix.

[0077] It should be noted that when splitting the spectrum matrix into high-light and low-light matrices, the empirically set threshold scheme mentioned above is not necessarily used. Filtering or other methods can also be used, as long as the harmonic structure can be clearly and completely separated.

[0078] Step S133: Construct the actual spectral energy high-density subspace matrix using all elements that are greater than or equal to the threshold.

[0079] In this embodiment of the application, the speech synthesis device splits the highlight matrix, that is, matrix A normed Set all elements in A that are less than the threshold THR to 0. normed Generate highlight matrix A high The function can be expressed as:

[0080]

[0081] Step S134: Construct the actual spectral energy low-density subspace matrix using all elements that are less than the threshold.

[0082] In this embodiment of the application, the speech synthesis device splits the low-light matrix, that is, matrix A normed All elements in A that are greater than or equal to the threshold THR should be set to 0. normed Generate low-light matrix A low The function can be expressed as:

[0083]

[0084] Step S14: Based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix, obtain the spectrum similarity loss value.

[0085] In this embodiment of the application, since there is a significant correlation between adjacent frames and between different harmonics in the spectrum, the correlation coefficient of the spectrum matrix can be used to represent their similarity. Therefore, the similarity loss L... similar It can be calculated in the following ways:

[0086]

[0087] Where a and b represent A respectivelyhigh B high The vectorized representation of ; Cov(a,b) is the covariance of a and b; σ a ,σ b Let A be the standard deviation of a and b; C is a very small positive real number. high B is the actual high-density subspace matrix of spectral energy. high To predict the high-density subspace matrix of spectral energy.

[0088] Specifically, this application also provides a matrix vectorization representation method, which expands the matrix row by row and concatenates them into a vector, as shown below:

[0089]

[0090] a = [a 11 ,a 12 ,...,a 1T ,a 21 ,a 22 ,...,a 2T ,...,a D1 ,a D2 ,...,a DT ]

[0091] Where D is the spectral feature dimension and T is the spectral length (i.e., the number of speech frames).

[0092] Why must the spectral similarity loss be calculated on the highlight matrix instead of directly on the original spectral matrix? Because the main components of speech, such as content and timbre, are represented in the spectrogram as the fundamental frequency, harmonics, and the relative distances between harmonics. The goal of spectral similarity loss is to quantify the differences between these main elements in two spectrograms. When audio contains noise, these main elements become mixed with noise energy in the spectrogram and are difficult to distinguish, especially in the early stages of model training when the generated spectrogram is very blurry. Therefore, the spectral similarity loss needs to be calculated on the highlight matrix.

[0093] Step S15: Obtain the energy distribution loss value based on the actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix.

[0094] In this embodiment of the application, if for all elements of the spectrum matrix, i.e. matrix A normedCalculating the energy distribution on the low-light matrix will significantly affect the results, particularly the bright harmonic structure. To enable the speech synthesis model to generate more accurate speech quality details, the energy distribution loss is calculated only on the low-light matrix. Specifically, the algorithm for the energy distribution loss function provided in this application calculates the KL divergence (relative entropy) in blocks of the spectral matrix and then averages the results.

[0095] Specifically, the calculation process for the energy distribution loss value is as follows:

[0096] (1) Submatrix splitting. The low-light matrix obtained by splitting the Mel spectrum matrix of the real speech into submatrixes is denoted as A. low The low-light matrix obtained from the Mel spectrum matrix predicted by the model is denoted as B. low . A low B low Divide into multiple submatrices of size M×N, padding the edges with 0s if necessary. Let the submatrices be denoted as... and i∈[1,L]. M×N can be set empirically, such as 10×8, where L is the number of submatrices.

[0097] (2) Calculate the distribution function of the submatrix. Assuming that the elements in each submatrix follow a Gaussian distribution, their mean and variance can be estimated to obtain their probability density function. The probability density function is denoted as submatrix The probability density function is denoted as

[0098] (3) Calculate the energy distribution loss. For each pair of submatrices with the same position, calculate their KL divergence and then take the average value. Use this average value as the energy distribution loss. That is:

[0099]

[0100] Where D KL Let L be the KL divergence, and L be the number of submatrices.

[0101] In other implementations, since calculating the KL divergence for all submatrix pairs is computationally expensive, random sampling of submatrix pairs can be used instead of full calculation in actual model training. When there are enough training data samples and enough training iterations, the performance of random sampling and full calculation is comparable.

[0102] Specifically, the speech synthesis device first sets a sampling rate r, such as r = 0.1. Based on the sampling rate, the number of sampled submatrices L′ = rL is calculated. For each sampled submatrix, A... low Blow The sampling position is random, as long as the submatrix does not exceed the boundary. That is, the coordinates of the starting position (x,y) = (rand(1,DM),rand(1,TN)), where rand(a,b) means randomly selecting an integer within the integer range [a,b].

[0103] Step S16: Train the speech synthesis model according to the spectral similarity loss value and the energy distribution loss value.

[0104] In this embodiment of the application, after obtaining the spectral similarity loss L similar and energy distribution loss L distrib Subsequently, the spectral loss function for training the speech synthesis model proposed in this application can be specifically expressed as:

[0105] L = w1·MAE(A,B) + w2·L similar (A high B high )+w3·L distrib (A low B low )

[0106] w1, w2, and w3 are the weights of each sub-item, which default to 1 and can be adjusted based on experience.

[0107] The speech synthesis device inputs spoken text into the speech synthesis model to be trained, and obtains predicted acoustic features; it then obtains the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features, as well as the predicted spectral energy high-density subspace matrix and the predicted spectral energy low-density subspace matrix of the predicted acoustic features; based on the actual spectral energy high-density subspace matrix and the predicted spectral energy high-density subspace matrix, it obtains a spectral similarity loss value; based on the actual spectral energy low-density subspace matrix and the predicted spectral energy low-density subspace matrix, it obtains an energy distribution loss value; and it trains the speech synthesis model according to the spectral similarity loss value and the energy distribution loss value. Through this method, the speech synthesis device trains the speech synthesis model using the spectral similarity loss function and the energy distribution loss function to alleviate the problems of overall blandness and local flaws in the synthesized speech, thereby improving the speech synthesis effect.

[0108] This application proposes a spectral similarity loss to evaluate the difference in harmonic structure between the model-generated spectrum and the real speech spectrum; it also proposes an energy distribution loss to evaluate the difference in timbre details between the model-generated spectrum and the real speech spectrum; and it proposes an improved scheme for random sampling submatrix, which greatly reduces the amount of computation without affecting the model training results.

[0109] based on Figure 1For the training method of the speech synthesis model shown, please refer to [link / reference needed]. Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of the speech synthesis method provided in this application.

[0110] Specifically, such as Figure 4 As shown, the speech synthesis model training method of this application embodiment specifically includes the following steps:

[0111] Step S21: Obtain the text to be synthesized speech.

[0112] Step S22: Input the speech text to be synthesized into the pre-trained speech synthesis model to obtain the corresponding audio text.

[0113] In this embodiment of the application, the pre-trained speech synthesis model is obtained through... Figure 1 The model was trained using the method shown, and the functionality of the related speech synthesis model was already demonstrated during the training process, so it will not be elaborated further here.

[0114] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0115] To implement the speech synthesis model training method and speech synthesis method described in the above embodiments, this application also proposes a speech synthesis apparatus, which can be found in detail below. Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the speech synthesis device provided in this application.

[0116] The speech synthesis apparatus 300 of this application embodiment includes a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled together.

[0117] The memory 31 is used to store program data, and the processor 32 is used to execute the program data to implement the speech synthesis model training method and speech synthesis method described in the above embodiments.

[0118] In this embodiment, processor 32 can also be referred to as a CPU (Central Processing Unit). Processor 32 may be an integrated circuit chip with signal processing capabilities. Processor 32 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 32 can be any conventional processor.

[0119] To implement the speech synthesis model training method and speech synthesis method of the above embodiments, this application also provides a computer-readable storage medium, such as... Figure 6 As shown, the computer-readable storage medium 400 is used to store program data 41, which, when executed by the processor, is used to implement the speech synthesis model training method and the speech synthesis method as described in the above embodiments.

[0120] This application also provides a computer program product, wherein the computer program product includes a computer program operable to cause a computer to execute the speech synthesis model training method and the speech synthesis method as described in the embodiments of this application. The computer program product can be a software installation package.

[0121] The speech synthesis model training method and speech synthesis method described in the above embodiments of this application, when implemented as software functional units and sold or used as independent products, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for training a speech synthesis model, characterized in that, The training method for the speech synthesis model includes: Input the spoken text into the speech synthesis model to be trained to obtain the predicted acoustic features; Obtain the actual acoustic features corresponding to the spoken text; Obtain the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features, as well as the predicted spectral energy high-density subspace matrix and the predicted spectral energy low-density subspace matrix of the predicted acoustic features; Based on the actual spectrum energy high-density subspace matrix and the predicted spectrum energy high-density subspace matrix, obtain the spectrum similarity loss value; Based on the actual spectrum energy low-density subspace matrix and the predicted spectrum energy low-density subspace matrix, the energy distribution loss value is obtained; The speech synthesis model is trained according to the spectral similarity loss value and the energy distribution loss value.

2. The model training method according to claim 1, characterized in that, The process of obtaining the actual spectral energy high-density subspace matrix and the actual spectral energy low-density subspace matrix of the actual acoustic features includes: Obtain the spectral matrix of the actual acoustic features; Calculate the average value of all elements in the spectrum matrix; Set the threshold according to the average value; The actual spectral energy high-density subspace matrix is ​​constructed using all elements that are greater than or equal to the threshold. The actual spectral energy low-density subspace matrix is ​​constructed using all elements that are less than the threshold.

3. The model training method according to claim 2, characterized in that, After obtaining the spectral matrix of the actual acoustic features, the model training method further includes: Obtain the minimum and maximum elements in the spectrum matrix; The minimum and maximum elements are used to normalize all elements of the spectrum matrix to obtain the normalized spectrum matrix.

4. The model training method according to claim 1, characterized in that, The step of obtaining the spectral similarity loss value based on the actual spectral energy high-density subspace matrix and the predicted spectral energy high-density subspace matrix includes: Obtain the actual high-density vector of the actual spectral energy high-density subspace matrix; Obtain the predicted high-density vector of the predicted spectral energy high-density subspace matrix; The spectral similarity loss value is obtained by using the covariance and standard deviation of the actual high-density vector and the predicted high-density vector.

5. The model training method according to claim 4, characterized in that, The columns of the actual spectral energy high-density subspace matrix are divided according to the spectral length, and the rows are divided according to the spectral feature dimension; The step of obtaining the actual high-density vector of the actual spectral energy high-density subspace matrix includes: The actual spectral energy high-density subspace matrix is ​​expanded along the row direction and spliced ​​into a one-dimensional actual high-density vector.

6. The model training method according to claim 1, characterized in that, The step of obtaining the energy distribution loss value based on the actual spectral energy low-density subspace matrix and the predicted spectral energy low-density subspace matrix includes: The actual spectral energy low-density subspace matrix is ​​divided into several actual low-density submatrices, and the predicted spectral energy low-density subspace matrix is ​​divided into the same number of predicted low-density submatrices. Obtain the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position; The energy distribution loss value is calculated as the average of all relative entropies.

7. The model training method according to claim 6, characterized in that: The process of obtaining the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position includes: Obtain the actual mean and actual variance of all elements in the actual low-density submatrix, and obtain the actual probability density function of the actual low-density submatrix based on the actual mean and actual variance. Obtain the predicted mean and predicted variance of all elements in the predicted low-density submatrix, and obtain the predicted probability density function of the predicted low-density submatrix based on the predicted mean and predicted variance. Using the actual probability density function and the predicted probability density function, the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position is determined.

8. The model training method according to claim 6, characterized in that, The process of obtaining the relative entropy of the actual low-density submatrix and the predicted low-density submatrix at the same position includes: Obtain a preset sampling rate, and determine the number of samples based on the preset sampling rate and the actual number of low-density submatrices; According to the sampling quantity, a random actual low-density submatrix and a random predicted low-density submatrix at the corresponding position are randomly obtained from the plurality of actual low-density submatrices. Obtain the relative entropy of random actual low-density submatrices and random predicted low-density submatrices at the same position.

9. A speech synthesis method, characterized in that, The speech synthesis method includes: Obtain the text to be synthesized speech; The speech text to be synthesized is input into a pre-trained speech synthesis model to obtain the speech audio corresponding to the speech text; The speech synthesis model is obtained by training the model training method according to any one of claims 1 to 8.

10. A speech synthesis device, characterized in that, The speech synthesis device includes a processor and a memory, the memory storing program data, and the processor executing the program data to implement the model training method as described in any one of claims 1-8, and / or the speech synthesis method as described in claim 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program data, which, when executed by a processor, is used to implement the model training method according to any one of claims 1-8, and / or the speech synthesis method according to claim 9.

Images