Related methods and related devices and equipment for training speech synthesis models

By acquiring and processing the standard and predicting feature spectrum errors of the speech synthesis model, the speech synthesis model is trained, and the problem of poor speech cloning effect in the prior art is solved, and the similarity and accuracy of speech synthesis are improved.

CN114203151BActive Publication Date: 2024-11-26GUANGZHOU HUYA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111272083.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-29
Publication Date
2024-11-26
Estimated Expiration
2041-10-29

AI Technical Summary

Technical Problem

Existing speech synthesis technologies are difficult to improve the effect of speech cloning, resulting in low similarity between speech synthesis.

Method used

By obtaining the difference between the standard feature spectrum and the predicted feature spectrum of the sample audio, the initial model is trained using these errors to form a speech synthesis model, thereby improving the similarity of speech synthesis. The specific steps include obtaining the feature information of the sample audio, performing Mel spectral and linear spectral processing, using a deep learning network to extract timbre features, combining the encoder and attention network to generate predicted feature spectrum, and using standard feature spectrum errors and predicted feature spectrums. The error is performed for model training.

Benefits of technology

The speech synthesis accuracy and similarity of the speech synthesis model are improved, so that the change trend of the predicted feature spectrum is consistent with the standard feature spectrum change trend of the sample audio, achieving higher speech synthesis similarity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114203151B_ABST
    Figure CN114203151B_ABST
Patent Text Reader

Abstract

The present application discloses related methods and related devices and equipment for training a speech synthesis model. The training method of the speech synthesis model includes: obtaining sample audio and its feature information, as well as the standard feature spectrum of each frame of sample audio; predicting the feature spectrum of the sample audio through the initial model to obtain the predicted feature spectrum of each frame of sample audio; performing difference processing on the standard feature spectrum of each adjacent frame and the predicted feature spectrum of each adjacent frame in turn to obtain the standard feature spectrum error and the predicted feature spectrum error; using the standard feature spectrum error and the predicted feature spectrum error to train the initial model to obtain the speech synthesis model. The above scheme trains the initial model by using the feature spectrum error after difference processing, so that the change trend of the predicted feature spectrum predicted by the speech synthesis model can be consistent with the change trend of the standard feature spectrum, thereby improving the prediction accuracy of the speech synthesis model, and then improving the similarity of speech synthesis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of speech synthesis, and in particular to related methods, devices and equipment for training speech synthesis models. Background Art

[0002] With the continuous development of electronic information processing technology, voice, as an important carrier for people to obtain information, has been widely used in daily life and work. Application scenarios involving voice usually include speech synthesis processing, which refers to synthesizing specified text or audio into target audio that meets the requirements.

[0003] Among them, voice cloning has received increasing attention in certain specific occasions with the widespread application of speech synthesis, such as AI intelligent calling systems and voice synthesis of specific people.

[0004] Based on this, how to improve the effect of speech cloning in speech synthesis, that is, how to improve the similarity of speech synthesis, has become a technical problem that needs to be solved at present. Summary of the invention

[0005] The present application provides a method and apparatus for training a speech synthesis model, which can improve the similarity of speech synthesis.

[0006] The present application provides a method for training a speech synthesis model, the method comprising: obtaining sample audio, characteristic information of the sample audio, and a standard characteristic spectrum of each frame of the sample audio; predicting the characteristic spectrum of the sample audio based on the sample audio and its characteristic information through an initial model to obtain a predicted characteristic spectrum of each frame of the sample audio; performing a subtraction process on the standard characteristic spectrum of each frame of the sample audio from the standard characteristic spectrum of the next frame to obtain a standard characteristic spectrum error, and performing a subtraction process on the predicted characteristic spectrum of each frame of the sample audio from the predicted characteristic spectrum of the next frame to obtain a predicted characteristic spectrum error; training the initial model using the standard characteristic spectrum error and the predicted characteristic spectrum error, and determining the trained model as the speech synthesis model.

[0007] The standard characteristic spectrum includes: a standard Mel spectrum and a standard linear spectrum; the predicted characteristic spectrum includes: a predicted Mel spectrum and a predicted linear spectrum; the standard characteristic spectrum of each frame of sample audio is subtracted from the standard characteristic spectrum of the next frame in turn to obtain a standard characteristic spectrum error, and the predicted characteristic spectrum of each frame of sample audio is subtracted from the predicted characteristic spectrum of the next frame to obtain the predicted characteristic spectrum error, the steps of: subtracting the Mel frequency of each frame in the standard Mel spectrum from the Mel frequency of the next frame in turn to obtain a standard Mel spectrum error between the Mel frequencies of adjacent frames; and subtracting the Mel frequency of each frame in the predicted Mel spectrum from the Mel frequency of the next frame in turn to obtain a standard Mel spectrum error between the Mel frequencies of adjacent frames. The predicted Mel spectrum error; and the frequency of each frame in the standard linear spectrum is subtracted from the frequency of the next frame in turn to obtain the standard linear spectrum error between the frequencies of adjacent frames; and the frequency of each frame in the predicted linear spectrum is subtracted from the frequency of the next frame in turn to obtain the predicted linear spectrum error between the frequencies of adjacent frames; the initial model is trained using the standard feature spectrum error and the predicted feature spectrum error, and the step of determining the trained model as a speech synthesis model includes: in response to the similarity between the standard Mel spectrum error and the predicted Mel spectrum error and / or the similarity between the standard linear spectrum error and the predicted linear spectrum error meeting the preset requirements, the training is completed and the speech synthesis model is obtained.

[0008] Among them, the step of predicting the characteristic spectrum of the sample audio based on the sample audio and its characteristic information through the initial model to obtain the predicted characteristic spectrum of each frame of sample audio also includes: performing convolution processing on the predicted Mel spectrum through the initial model; performing residual processing based on the predicted Mel spectrum before and after the convolution processing to obtain a Mel residual spectrum; subtracting the standard characteristic spectrum of each frame of sample audio from the standard characteristic spectrum of the next frame in turn to obtain a standard characteristic spectrum error, and the step of subtracting the predicted characteristic spectrum of each frame of sample audio from the predicted characteristic spectrum of the next frame to obtain the predicted characteristic spectrum error also includes: subtracting the frequency of each frame in the Mel residual spectrum from the frequency of the next frame in turn to obtain a predicted residual spectrum error between the frequencies of adjacent frames; using the standard characteristic spectrum error and the predicted characteristic spectrum error to train the initial model, and the step of determining the trained model as a speech synthesis model also includes: in response to the similarity between the standard Mel spectrum error and the predicted Mel spectrum error and / or the similarity between the standard linear spectrum error and the predicted linear spectrum error and / or the similarity between the predicted residual spectrum error and the standard Mel spectrum error meeting the preset requirements, then obtaining the speech synthesis model.

[0009] Among them, the steps of obtaining sample audio, obtaining feature information of the sample audio, and a standard feature spectrum of each frame of sample audio include: obtaining sample audio, and obtaining the phonemes, rhythm and pitch of the sample audio, and using the phonemes, rhythm and pitch as feature information; performing frame windowing and Fourier transform on the sample audio to obtain a standard linear spectrum of the sample audio; inputting the standard linear spectrum into a Mel filter group for filtering processing to obtain a standard Mel spectrum.

[0010] Among them, the training method of the speech synthesis model also includes: inputting the standard Mel spectrum of the sample audio into the deep learning network, and using the deep learning network to extract the timbre feature matrix of the sample audio; extracting audio features based on the sample audio to obtain the audio feature matrix of the sample audio.

[0011] The initial model includes an encoder, an attention network, and a decoder that are cascaded to each other; the initial model predicts the characteristic spectrum of the sample audio based on the sample audio and its characteristic information to obtain the predicted characteristic spectrum of each frame of the sample audio, including: inputting the phonemes, rhythm, and pitch of the sample audio into the encoder and the attention network in the initial model for encoding to obtain encoded data;

[0012] Among them, the step of training the initial model using the standard feature spectrum error and the predicted feature spectrum error, and determining the trained model as the speech synthesis model also includes: determining the mean square error between each frame in the standard feature spectrum and the corresponding frame in the predicted feature spectrum based on the standard feature spectrum and the predicted feature spectrum; training the initial model using the mean square error between each frame in the standard feature spectrum and the corresponding frame in the predicted feature spectrum, and determining the trained model as the speech synthesis model.

[0013] The present application also provides a speech synthesis method, which includes: acquiring feature information of an object to be synthesized and a target object; inputting the feature information of the object to be synthesized and the target object into a speech synthesis model for prediction, and obtaining a predicted feature spectrum synthesized by the speech synthesis model; outputting synthesized speech based on the predicted feature spectrum through the speech synthesis model; wherein the speech synthesis model is trained by any of the above-mentioned speech synthesis model training methods.

[0014] The present application also provides a training device for a speech synthesis model, which includes: an acquisition module, which is used to acquire sample audio, characteristic information of the sample audio, and a standard characteristic spectrum of each frame of sample audio; a prediction module, which is used to predict the characteristic spectrum of the sample audio based on the sample audio and its characteristic information through an initial model, and obtain a predicted characteristic spectrum of each frame of sample audio; a calculation module, which is used to sequentially perform a difference processing on the standard characteristic spectrum of each frame of sample audio with the standard characteristic spectrum of the next frame, and obtain a standard characteristic spectrum error, and perform a difference processing on the predicted characteristic spectrum of each frame of sample audio with the predicted characteristic spectrum of the next frame, and obtain a predicted characteristic spectrum error; a training module, which is used to train the initial model using the standard characteristic spectrum error and the predicted characteristic spectrum error, and determine the trained model as the speech synthesis model.

[0015] The present application also provides a speech synthesis device, which includes: an acquisition module for acquiring feature information of an object to be synthesized and a target object; a prediction module for inputting the feature information of the object to be synthesized and the target object into a speech synthesis model for prediction, and obtaining a predicted feature spectrum synthesized by the speech synthesis model; an output module for outputting synthesized speech based on the predicted feature spectrum through the speech synthesis model; wherein the speech synthesis model is trained by any of the above-mentioned speech synthesis model training methods.

[0016] The present application also provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement a training method and a speech synthesis method for any of the above-mentioned speech synthesis models.

[0017] The present application also provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the training method and speech synthesis method of any of the above-mentioned speech synthesis models are implemented.

[0018] The above scheme, the present application obtains the standard feature spectrum of each frame of sample audio, and then predicts the feature spectrum of the sample audio based on the sample audio and its feature information through the initial model to obtain the predicted feature spectrum of each frame of sample audio, and calculates the standard feature spectrum error and the predicted feature spectrum error in turn, and finally uses the standard feature spectrum error and the predicted feature spectrum error to train the initial model, and determines the trained model as a speech synthesis model, which can be trained based on the predicted feature spectrum error between adjacent frames in the predicted feature spectrum and the standard feature spectrum error between adjacent frames in the standard feature spectrum, so that the change trend of the predicted feature spectrum predicted by the speech synthesis model after training can be consistent with the change trend of the standard feature spectrum of the sample audio, thereby improving the speech synthesis accuracy of the speech synthesis model and improving the similarity of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of an embodiment of a training method for a speech synthesis model of the present application;

[0020] Figure 2 It is a flowchart of another embodiment of the training method of the speech synthesis model of the present application;

[0021] Figure 3 It is a flowchart of an embodiment of the speech synthesis method of the present application;

[0022] Figure 4 It is a schematic diagram of the framework of an embodiment of a training device for speech synthesis of the present application;

[0023] Figure 5 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application;

[0024] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0025] Figure 7 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects. There can be three kinds of relationships. For example, A and / or B can be: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally means that the associated objects before and after are in an "or" relationship. In addition, "more" in this article means two or more than two.

[0029] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of a training method for a speech synthesis model of the present application.

[0030] Step S11: Acquire sample audio, and acquire feature information of the sample audio, and a standard feature spectrum of each frame of the sample audio.

[0031] The sample audio is obtained, and based on the sample audio, feature information of the sample audio and a standard feature spectrum of each frame of the sample audio are obtained.

[0032] Among them, the characteristic information of the sample audio may include one or more of the characteristic information of the sample audio such as phonemes, pitch, phoneme duration, sound intensity, rhythm, etc. The standard characteristic spectrum may include one or more of the characteristic spectra of the sample audio such as Mel spectrum, linear spectrum, linear prediction coefficient (LPC), linear prediction cepstrum coefficient (LPCC), discrete wavelet transform (DWT), perceptual linear prediction (PLP), etc. Among them, the characteristic spectrum of this embodiment may include relevant characteristic spectra reflecting the timbre, rhythm or other audio features that need to be restored of the sample audio, which can be selected based on actual needs. For example: when the trained speech synthesis model needs to restore the timbre of the object to be synthesized, the type of characteristic spectrum selected in the training process here can at least include Mel spectrum. Among them, the timbre generation of the synthesized speech mainly depends on the Mel spectrum and its related features. When the trained speech synthesis model needs to restore other features of the object to be synthesized, the type of characteristic spectrum selected in the training process here can select the corresponding characteristic spectrum.

[0033] In a specific application scenario, feature information of the sample audio can be obtained based on the sample audio through manual or pre-trained deep learning models.

[0034] In a specific application scenario, the standard characteristic spectrum of each frame of sample audio can be obtained based on the sample audio through the corresponding pre-trained deep learning model. In another specific application scenario, the standard characteristic spectrum of each frame of sample audio can also be obtained through audio processing tools such as Aubio tools, Essentia tools, or Librosa tools. The specific method of obtaining the standard characteristic spectrum of each frame of sample audio is not limited here.

[0035] Step S12: predicting the characteristic spectrum of the sample audio based on the sample audio and its characteristic information through the initial model to obtain the predicted characteristic spectrum of each frame of the sample audio.

[0036] After acquiring the characteristic information of the sample audio, the sample audio and its characteristic information are input into the initial model, and the characteristic spectrum of the sample audio is predicted based on the sample audio and its characteristic information by the initial model to obtain the predicted characteristic spectrum of each frame of the sample audio.

[0037] The type of the predicted characteristic spectrum corresponds to the type of the standard characteristic spectrum in step S11.

[0038] Step S13: Subtract the standard characteristic spectrum of each frame of sample audio from the standard characteristic spectrum of the next frame to obtain a standard characteristic spectrum error, and subtract the predicted characteristic spectrum of each frame of sample audio from the predicted characteristic spectrum of the next frame to obtain a predicted characteristic spectrum error.

[0039] The standard characteristic spectrum of each frame of sample audio is subtracted from the standard characteristic spectrum of the next frame to obtain the standard characteristic spectrum error. And the predicted characteristic spectrum of each frame of sample audio is subtracted from the predicted characteristic spectrum of the next frame to obtain the predicted characteristic spectrum error.

[0040] In a specific application scenario, assume that there are 10 frames of standard characteristic spectra of sample audio: A1, A2, A3, A4, A5, A6, A7, A8, A9 and A10. Then, the standard characteristic spectrum of each frame of sample audio is subtracted from the standard characteristic spectrum of the next frame in turn, and the standard characteristic spectrum error is: S 1-2 , S 2-3 , S 3-4 , S 4-5 , S 5-6 , S 6-7 , S 7-8 , S 8-9 , S 9-10 .

[0041] In a specific application scenario, assume that there are 10 frames of predicted characteristic spectra of sample audio: B1, B2, B3, B4, B5, B6, B7, B8, B9 and B10. Then, the predicted characteristic spectrum of each frame of sample audio is subtracted from the predicted characteristic spectrum of the next frame in turn, and the predicted characteristic spectrum error is: H 1-2 , H 2-3 , H 3-4 , H 4-5 , H 5-6 , H 6-7 , H 7-8 , H 8-9 , H 9-10 .

[0042] Step S14: train the initial model using the standard feature spectrum error and the predicted feature spectrum error, and determine the trained model as the speech synthesis model.

[0043] After obtaining the standard characteristic spectrum error of the sample audio and the predicted characteristic spectrum error predicted based on the initial model, the initial model is trained using the standard characteristic spectrum error and the predicted characteristic spectrum error, and the trained model is determined as the speech synthesis model.

[0044] In a specific application scenario, the initial model can be trained based on the standard feature spectrum error and the predicted feature spectrum error through a loss function until the loss function converges to obtain a trained speech synthesis model. The loss function may include a cross entropy loss function, an absolute value loss function, a square loss function, etc., which are not specifically limited here.

[0045] In a specific application scenario, the similarity between the standard feature spectrum error and the predicted feature spectrum error can be calculated. In response to the similarity between the standard feature spectrum error and the predicted feature spectrum error satisfying the similarity threshold, the training is completed and a trained speech synthesis model is obtained.

[0046] Through the above steps, the training method of the speech synthesis model of the present embodiment obtains the standard feature spectrum of each frame of sample audio, and then predicts the feature spectrum of the sample audio based on the sample audio and its feature information through the initial model to obtain the predicted feature spectrum of each frame of sample audio, and calculates the standard feature spectrum error and the predicted feature spectrum error in turn. Finally, the standard feature spectrum error and the predicted feature spectrum error are used to train the initial model, and the trained model is determined as the speech synthesis model. It can be trained based on the predicted feature spectrum error between adjacent frames in the predicted feature spectrum and the standard feature spectrum error between adjacent frames in the standard feature spectrum, so that the change trend of the predicted feature spectrum predicted by the speech synthesis model after the training is consistent with the change trend of the standard feature spectrum of the sample audio, thereby improving the speech synthesis accuracy of the speech synthesis model and improving the similarity of speech synthesis.

[0047] See also Figure 2 , Figure 2 It is a flowchart of another embodiment of the training method of the speech synthesis model of the present application.

[0048] Step S21: Obtain sample audio, and obtain the phonemes, rhythm and pitch of the sample audio, use the phonemes, rhythm and pitch as feature information, frame and window the sample audio and perform Fourier transform to obtain a standard linear spectrum of the sample audio, input the standard linear spectrum into a Mel filter group for filtering, and obtain a standard Mel spectrum.

[0049] The sample audio is obtained, and the phonemes, rhythm and pitch of the sample audio are obtained. Among them, the phoneme is the smallest unit of speech divided according to the natural attributes of speech. Pitch refers to various sounds with different pitches, that is, the height of the sound, a basic feature of the sound. The phoneme duration is the pronunciation duration of the corresponding phoneme. Rhythm includes the level and oblique format and rhyme rules of the sample audio.

[0050] In a specific application scenario, multiple recorders with different timbres can record sample audio based on the required phonemes, rhythms, and pitches. The sample audio can cover most or all of the pitch range and phoneme duration range. All the phonemes, rhythms, and pitches of the sample audio are then extracted from the sample audio.

[0051] In a specific application scenario, when obtaining the phonemes, rhythm and pitch of the sample audio, the sample audio can be input into a pre-trained deep learning model for prediction to obtain the phonemes, rhythm and pitch of the sample audio. In another specific application scenario, when obtaining the phonemes, rhythm and pitch of the sample audio, the phonemes of the sample audio can be analyzed based on the dictionary corresponding to the sample audio, the pitch of the sample audio can be analyzed based on the international standard sound, and the sample audio can be predicted through a pre-trained deep learning model to obtain the rhythm of the sample audio. In another specific application scenario, when obtaining the phonemes, rhythm and pitch of the sample audio, manual analysis of the sample audio can be received to obtain the phonemes, rhythm and pitch of the sample audio.

[0052] Among them, using the phonemes, rhythm and pitch of the sample audio as the basis for speech synthesis can enrich the features of the sample audio and improve the feature accuracy of the sample audio, thereby enabling speech synthesis to be more complete and accurate based on richer features, thereby improving the effect of speech synthesis.

[0053] This step also performs frame windowing and Fourier transformation on the sample audio to obtain a standard linear spectrum of the sample audio, and then inputs the standard linear spectrum into a Mel filter bank for filtering to obtain a standard Mel spectrum. Among them, the Fourier transform includes a short-time Fourier transform (STFT) or other Fourier transforms. Specifically, after the sample audio is framed, windowed and Fourier transformed, a time-frequency domain signal of the sample audio can be obtained, and then a standard linear spectrum of the sample audio is generated based on the time-frequency domain signal of the sample audio.

[0054] In a specific application scenario, the sample audio can be framed, windowed, and Fourier transformed to first obtain the time-frequency domain signal of the sample audio, and then take the amplitude of the time-frequency domain signal to generate a standard linear spectrum of the sample audio, so as to refine the features in the standard linear spectrum to improve the extraction speed, and then input the standard linear spectrum into the Mel filter group for filtering to obtain the standard Mel spectrum.

[0055] Step S22: Input the standard Mel spectrum of the sample audio into the deep learning network, and use the deep learning network to extract the timbre feature matrix of the sample audio. Perform audio feature extraction based on the sample audio to obtain the audio feature matrix of the sample audio.

[0056] The standard Mel spectrum of the sample audio is input into the deep learning network, and the timbre feature matrix of the sample audio is extracted by the deep learning network, so that the model can learn the timbre of the sample audio through the timbre feature matrix. Audio feature extraction is performed based on the sample audio to obtain the audio feature matrix of the sample audio, so that the model can learn the rhythm of the sample audio through the audio feature matrix.

[0057] In a specific application scenario, the rhythm of the sample audio can be extracted through the global style token (GST) to obtain the audio feature matrix of the sample audio. In another specific application scenario, the rhythm of the sample audio can also be extracted through the reference_encoder or the unsupervised variational autoencoder (VAE) to obtain the audio feature matrix of the sample audio.

[0058] In a specific application scenario, this step can also input the standard Mel spectrum of the sample audio into the deep learning network, and use the deep learning network to extract the timbre feature matrix of the sample audio. Finally, the Mel spectrum is input into the deep learning network to extract the timbre information of the audio to be synthesized, so as to extract the timbre feature matrix of the sample audio using the deep learning network.

[0059] Among them, the deep learning network for extracting the timbre feature matrix in this step can also include a combination network of a bidirectional gated recurrent unit (GRU) network + a fully connected network (Fully Connected Layers, FC) + a Bayesian network (Bayesian network, BN), a long short-term memory model, a convolutional neural network or a recurrent neural network and other deep learning networks, and the specific type can be set based on the actual situation and is not limited here. And the deep learning network in this step is a network that has been pre-trained for extracting the timbre feature matrix based on Mel spectrum and can be applied.

[0060] This embodiment improves the accuracy and precision of the predicted feature spectrum generated by the initial model by adding an audio feature matrix produced based on real sample audio to the decoded input.

[0061] Step S23: Input the phonemes, rhythm and pitch of the sample audio into the encoder and attention network in the initial model for encoding to obtain encoded data, and input the encoded data, the audio feature matrix and timbre feature matrix of the sample audio into the decoder in the initial model for decoding to obtain a predicted feature spectrum.

[0062] After obtaining the phonemes, rhythm and pitch of the sample audio, the phonemes, rhythm and pitch of the sample audio are input into the encoder and attention network in the initial model for encoding to obtain encoded data.

[0063] The initial model of this embodiment is based on the tacotron1 model and improved. Specifically, the initial model of this embodiment includes a cascaded encoder, an attention network, and a decoder. Since the feature extraction of phonemes, rhythms, and pitches does not need to take into account contextual information, this embodiment removes the bidirectional GRU module in the tacotron1 model to simplify the initial model structure, reduce invalid model operations, and thus improve the speech synthesis efficiency of the initial model and the final speech synthesis model.

[0064] Furthermore, this embodiment replaces the Bahdanau Attention mechanism in the original tacotron1 model with a location-based attention mechanism (Location Based Attention), that is, the initial model adopts a location-based attention mechanism, wherein the location-based attention mechanism ignores or reduces silence in the speech data, thereby reducing the invalid analysis and consumption of silence in speech synthesis, thereby improving the efficiency and quality of speech synthesis. Furthermore, the initial model also adopts Forward Location Based Attention as an attention mechanism to further reduce the invalid analysis and consumption of silence, thereby improving the efficiency and quality of speech synthesis.

[0065] In a specific application scenario, the initial model can also add a Stop Token network to segment the decoded data through the StopToken network, thereby separating the entire segment of synthesized speech to make it conform to the pronunciation rules, thereby improving the quality of speech synthesis.

[0066] The phonemes, rhythm, and pitch of the sample audio are input into the encoder in the initial model for encoding, and then the output of the encoder is input into the attention network for processing, and finally the output of the attention network is used as the encoded data.

[0067] After obtaining the encoded data, the audio feature matrix, timbre feature matrix and encoded data are input into the decoder of the initial model for decoding to obtain the predicted feature spectrum. In this way, part of the real audio information is embedded in the decoding process of the initial model, so that the model can further learn the rhythm of the sample audio, thereby improving the speech synthesis similarity of the initial model and improving the accuracy and reliability of the predicted feature spectrum.

[0068] The standard characteristic spectrum in this embodiment may include: a standard Mel spectrum and a standard linear spectrum, and the predicted characteristic spectrum may include a predicted Mel spectrum and a predicted linear spectrum.

[0069] Step S24: Subtract the standard characteristic spectrum of each frame of sample audio from the standard characteristic spectrum of the next frame to obtain a standard characteristic spectrum error, and subtract the predicted characteristic spectrum of each frame of sample audio from the predicted characteristic spectrum of the next frame to obtain a predicted characteristic spectrum error.

[0070] Since the standard characteristic spectrum of this embodiment includes the standard Mel spectrum and the standard linear spectrum, and the predicted characteristic spectrum includes the predicted Mel spectrum and the predicted linear spectrum, the specific calculation of this step is:

[0071] The Mel frequency of each frame in the standard Mel spectrum is subtracted from the Mel frequency of the next frame in turn to obtain the standard Mel spectrum error between the Mel frequencies of adjacent frames; and the Mel frequency of each frame in the predicted Mel spectrum is subtracted from the Mel frequency of the next frame in turn to obtain the predicted Mel spectrum error between the Mel frequencies of adjacent frames.

[0072] And the frequency of each frame in the standard linear spectrum is subtracted from the frequency of the next frame in turn to obtain the standard linear spectrum error between the frequencies of adjacent frames; and the frequency of each frame in the predicted linear spectrum is subtracted from the frequency of the next frame in turn to obtain the predicted linear spectrum error between the frequencies of adjacent frames.

[0073] At this point, this step obtains the standard Mel spectrum error, the predicted Mel spectrum error, the standard linear spectrum error, and the predicted linear spectrum error.

[0074] In a specific application scenario, this step can also input the predicted Mel spectrum into the 5 convolutional layers, tanh activation function and fully connected layer in the initial model in sequence for convolution processing, and perform residual processing on the output data after convolution processing and the predicted Mel spectrum to obtain a Mel residual spectrum.

[0075] After obtaining the Mel residual spectrum, the calculation in this step can further perform a difference process on the frequency of each frame in the Mel residual spectrum with the frequency of the next frame in turn to obtain the predicted residual spectrum error between the frequencies of each adjacent frame. This increases the number of error comparisons, improves the training labels in the model training process, further improves the model training effect, and improves the speech synthesis similarity of the final speech synthesis model.

[0076] This step further increases the training of Mel features in the model training process by adding the Mel residual spectrum and its error comparison, thereby improving the model's restoration of timbre, and further improving the restoration of timbre of the synthesized speech of the synthesis model.

[0077] Step S25: train the initial model using the standard feature spectrum error and the predicted feature spectrum error, and determine the trained model as the speech synthesis model.

[0078] The initial model is trained using the standard feature spectrum error and the predicted feature spectrum error, and the trained model is determined as the speech synthesis model.

[0079] In a specific application scenario, the initial model is trained using the standard Mel spectrum and the standard linear spectrum as well as the predicted Mel spectrum and the predicted linear spectrum, and in response to the similarity between the standard Mel spectrum error and the predicted Mel spectrum error and / or the similarity between the standard linear spectrum error and the predicted linear spectrum error meeting the preset requirements, a speech synthesis model is obtained. Among them, the preset requirements of this application scenario may include a similarity threshold or a similarity range, and the preset requirements may be specifically set based on the training means. When the similarity between the above errors reaches the similarity threshold or meets the similarity range, the training is completed and the speech synthesis model is obtained.

[0080] In a specific application scenario, the initial model is trained using the standard Mel spectrum and the standard linear spectrum as well as the predicted Mel spectrum and the predicted linear spectrum, and the gap between the standard Mel spectrum error and the predicted Mel spectrum error and / or the standard linear spectrum error and the predicted linear spectrum error is optimized by the ADAM optimizer until the preset requirements are met, and the training is completed to obtain a speech synthesis model. The preset requirements in this application scenario may include the optimization criteria of the ADAM optimizer.

[0081] In a specific application scenario, the initial model can also be trained using the standard Mel spectrum, standard linear spectrum, predicted Mel spectrum, predicted linear spectrum, and predicted residual spectrum errors, and in response to the similarity between the standard Mel spectrum error and the predicted Mel spectrum error and / or the similarity between the standard linear spectrum error and the predicted linear spectrum error and / or the similarity between the predicted residual spectrum error and the standard Mel spectrum error meeting the preset requirements, a speech synthesis model is obtained. The preset requirements may include a similarity threshold or a similarity range, and the preset requirements may be specifically set based on the training method. When the similarity between the above errors reaches the similarity threshold or meets the similarity range, the training is completed and the speech synthesis model is obtained.

[0082] In a specific application scenario, when training the model, when the initial model is trained by the standard feature spectrum error and the predicted feature spectrum error, the mean square error between each frame in the standard feature spectrum and the corresponding frame in the predicted feature spectrum can be further calculated, that is, the mean square error between the standard Mel spectrum and the predicted Mel spectrum and the mean square error between the standard linear spectrum and the predicted linear spectrum; and the initial model is trained using the mean square error between each frame in the standard feature spectrum and the corresponding frame in the predicted feature spectrum. That is, in this application scenario, the training convergence is based on the error between each feature spectrum error and the error between each feature spectrum itself, thereby refining the training requirements, so that the model can predict the characteristic frequencies of the sample audio and the change trend of the entire sample audio, thereby improving the similarity between the synthesized speech and the sample audio.

[0083] Since the sample audio of this embodiment covers most or all of the pitch range and phoneme range, after the training is completed and the speech synthesis model is obtained, the speech synthesis model only needs a small amount of phonemes, rhythms and pitches of the object to be synthesized to perform speech synthesis or even speech generation with the same timbre, and based on the setting of the timbre feature matrix and the audio feature matrix, the speech synthesis model can learn different timbre information based on different sample audios, thereby further restoring the synthesized speech corresponding to the timbre of each sample audio, so that in the application process of the speech synthesis model, speech synthesis of different phonemes, rhythms and pitches can be realized based on the timbre of the object to be synthesized, and the similarity of speech synthesis can be improved to achieve speech cloning.

[0084] In a specific application scenario, in order to prevent the model from overfitting, regularization may be performed on parts of the model, for example, regularization may be performed on the decoder and / or encoder of the model.

[0085] Through the above steps, the training method of the speech synthesis model of this embodiment can enrich the features of the sample audio by taking the phonemes, rhythm and pitch of the sample audio as the feature information of speech synthesis, improve the feature accuracy of the sample audio, and thus improve the speech synthesis effect. This embodiment extracts the audio feature matrix and the timbre feature matrix of the sample audio respectively, and decodes the above features together with the encoded data as the input of the decoder, so that the model can further learn the rhythm and timbre of the sample audio, thereby improving the speech synthesis similarity of the initial model and improving the accuracy and reliability of the predicted feature spectrum. Moreover, this embodiment also trains the initial model based on the standard Mel spectrum error and the predicted Mel spectrum error and / or the standard linear spectrum error and the predicted linear spectrum error and / or the predicted residual spectrum error and the standard Mel spectrum error, and trains the initial model based on the error between the feature spectra of multiple response timbre such as the mean square error between the standard Mel spectrum and the predicted Mel spectrum and the mean square error between the standard linear spectrum and the predicted linear spectrum, so as to further improve the training effect of the initial model through multi-layer training and improve the speech synthesis similarity of the speech synthesis model.

[0086] See also Figure 3 , Figure 3 It is a flowchart of an embodiment of the speech synthesis method of the present application.

[0087] Step S31: Acquire feature information of the object to be synthesized and the target object.

[0088] An object to be synthesized is obtained, wherein the object to be synthesized may include an object to be synthesized for which voice restoration needs to be performed.

[0089] The target object may include an object in text or audio form, and its characteristic information includes information such as phonemes, pitch, phoneme duration, sound intensity, and rhythm.

[0090] After acquiring the feature information of the object to be synthesized and the target object, speech synthesis is performed based on the object to be synthesized using the feature information of the target object, so that the synthesis of speech with various other feature information based on the object to be synthesized can be achieved by changing the feature information of the target object.

[0091] In a specific application scenario, speech synthesis can be performed based on the timbre of the object to be synthesized using the characteristic information of the target object, so that the synthesis of speech of various other contents can be achieved while cloning the timbre of the object to be synthesized by changing the characteristic information of the target object.

[0092] In another specific application scenario, speech synthesis can also be performed based on the rhythm of the object to be synthesized using the characteristic information of the target object, so that the synthesis of speech of various other contents can be achieved while cloning the rhythm of the object to be synthesized by changing the characteristic information of the target object.

[0093] Step S32: inputting feature information of the object to be synthesized and the target object into the speech synthesis model for prediction, and obtaining a predicted feature spectrum synthesized by the speech synthesis model.

[0094] After obtaining the object to be synthesized, the object to be synthesized is input into the speech synthesis model for prediction, and a predicted feature spectrum synthesized by the speech synthesis model is obtained.

[0095] The speech synthesis model of this embodiment is a speech synthesis model trained by the training method of the speech synthesis model of any of the above embodiments. It is trained based on the standard feature spectrum error and the predicted feature spectrum error, so that the predicted feature spectrum predicted by the speech synthesis model and its change trend can restore the characteristics of the object to be synthesized and match the characteristic information of the target object, thereby improving the accuracy and reliability of the synthesized speech finally output.

[0096] Step S33: Outputting synthesized speech based on the predicted feature spectrum through the speech synthesis model.

[0097] After the predicted feature spectrum is obtained, the speech synthesis model is used to output the synthesized speech based on the predicted feature spectrum, thereby completing the speech synthesis of the object to be synthesized.

[0098] Through the above steps, after acquiring the feature information of the object to be synthesized and the target object, the speech synthesis method of this embodiment inputs the feature information of the target object and the object to be synthesized into the speech synthesis model for prediction, obtains the predicted feature spectrum synthesized by the speech synthesis model, and finally outputs the synthesized speech based on the predicted feature spectrum through the speech synthesis model. Since the speech synthesis model of this embodiment is a speech synthesis model trained by the training method of the speech synthesis model of any of the above embodiments, speech synthesis through the speech synthesis model can obtain a predicted feature spectrum that is consistent with the standard feature spectrum of the speech to be synthesized, and then restore the features of the object to be synthesized and match the feature information of the target object, thereby improving the accuracy and similarity between the synthesized speech finally output and the object to be synthesized, and achieving the matching of the feature information of the synthesized speech and the target object.

[0099] In other embodiments, the speech synthesis model includes an encoder, an attention network, and a decoder that are cascaded with each other. The feature information of the object to be synthesized and the target object is input into the speech synthesis model for prediction, and the step of obtaining the predicted feature spectrum synthesized by the speech synthesis model can also be performed by encoding the feature information through the encoder and the attention network of the speech synthesis model to obtain the encoded data; the standard Mel spectrum of the object to be synthesized is input into the deep learning network, and the timbre feature matrix of the object to be synthesized is extracted by the deep learning network; audio features are extracted based on the object to be synthesized to obtain the audio feature matrix of the sample audio; and the decoder of the speech synthesis model predicts based on the feature information, the timbre feature matrix, and the audio feature matrix to obtain the predicted feature spectrum synthesized by the speech synthesis model. Among them, the specific method for obtaining the audio feature matrix and the timbre feature matrix, the type of the feature spectrum, etc. are the same as those in the aforementioned embodiment. Please refer to the previous text and will not be repeated here.

[0100] Among them, in the above steps, the characteristic information of the target object is predicted by the trained speech synthesis model, and the audio feature matrix and timbre feature matrix of the speech to be synthesized are extracted, so as to embed the audio information and timbre information of the real object to be synthesized, further improve the speech synthesis model to restore the timbre and rhythm of the object to be synthesized, further improve the restoration and cloning of the timbre of the object to be synthesized in the speech synthesis process, and then improve the similarity of speech synthesis.

[0101] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of an embodiment of the training device for speech synthesis of the present application. The training device 40 for the speech synthesis model includes an acquisition module 41, a prediction module 42, a calculation module 43 and a training module 44. The acquisition module 41 is used to acquire the sample audio, and acquire the feature information of the sample audio, and the standard feature spectrum of each frame of the sample audio; the prediction module 42 is used to predict the feature spectrum of the sample audio based on the sample audio and its feature information through the initial model, and obtain the predicted feature spectrum of each frame of the sample audio; the calculation module 43 is used to sequentially perform a difference process on the standard feature spectrum of each frame of the sample audio with the standard feature spectrum of the next frame, and obtain the standard feature spectrum error, and perform a difference process on the predicted feature spectrum of each frame of the sample audio with the predicted feature spectrum of the next frame, and obtain the predicted feature spectrum error; the training module 44 is used to train the initial model using the standard feature spectrum error and the predicted feature spectrum error, and determine the trained model as the speech synthesis model.

[0102] The calculation module 43 is also used to sequentially perform a subtraction process on the Mel frequency of each frame in the standard Mel spectrum and the Mel frequency of the next frame to obtain a standard Mel spectrum error between the Mel frequencies of adjacent frames; and sequentially perform a subtraction process on the Mel frequency of each frame in the predicted Mel spectrum and the next frame to obtain a predicted Mel spectrum error between the Mel frequencies of adjacent frames; and sequentially perform a subtraction process on the frequency of each frame in the standard linear spectrum and the next frame to obtain a standard linear spectrum error between the frequencies of adjacent frames; and sequentially perform a subtraction process on the frequency of each frame in the predicted linear spectrum and the next frame to obtain a predicted linear spectrum error between the frequencies of adjacent frames. The training module 44 is also used to complete the training and obtain a speech synthesis model in response to the similarity between the standard Mel spectrum error and the predicted Mel spectrum error and / or the similarity between the standard linear spectrum error and the predicted linear spectrum error meeting a preset requirement.

[0103] The prediction module 42 is also used to perform convolution processing on the predicted Mel spectrum through the initial model; perform residual processing based on the predicted Mel spectrum before and after the convolution processing to obtain a Mel residual spectrum. The calculation module 43 is also used to perform difference processing on the frequency of each frame in the Mel residual spectrum with the frequency of the next frame in turn to obtain a predicted residual spectrum error between the frequencies of each adjacent frame. The training module 44 is also used to obtain a speech synthesis model in response to the similarity between the standard Mel spectrum error and the predicted Mel spectrum error and / or the similarity between the standard linear spectrum error and the predicted linear spectrum error and / or the similarity between the predicted residual spectrum error and the standard Mel spectrum error meeting the preset requirements.

[0104] The acquisition module 41 is also used to acquire the sample audio, and acquire the phonemes, rhythm and pitch of the sample audio, and use the phonemes, rhythm and pitch as feature information; perform frame windowing and Fourier transform on the sample audio to obtain a standard linear spectrum of the sample audio; input the standard linear spectrum into the Mel filter group for filtering processing to obtain a standard Mel spectrum.

[0105] The acquisition module 41 is also used to input the standard Mel spectrum of the sample audio into the deep learning network, and use the deep learning network to extract the timbre feature matrix of the sample audio; perform audio feature extraction based on the sample audio to obtain the audio feature matrix of the sample audio.

[0106] The prediction module 42 is also used to input the phonemes, rhythm and pitch of the sample audio into the encoder and attention network in the initial model for encoding to obtain encoded data; and input the encoded data, the audio feature matrix and the timbre feature matrix of the sample audio into the decoder in the initial model for decoding to obtain a predicted feature spectrum.

[0107] The training module 44 is also used to determine the mean square error between each frame in the standard feature spectrum and the corresponding frame in the predicted feature spectrum based on the standard feature spectrum and the predicted feature spectrum; use the mean square error between each frame in the standard feature spectrum and the corresponding frame in the predicted feature spectrum to train the initial model, and determine the trained model as the speech synthesis model.

[0108] The above scheme can achieve the similarity of speech synthesis.

[0109] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application.

[0110] The speech synthesis device 50 includes: an acquisition module 51, which is used to acquire feature information of the object to be synthesized and the target object; a prediction module 52, which is used to input the feature information of the object to be synthesized and the target object into the speech synthesis model for prediction, and obtain a predicted feature spectrum synthesized by the speech synthesis model; an output module 53, which is used to output synthesized speech based on the predicted feature spectrum through the speech synthesis model;

[0111] The speech synthesis model is obtained by training using the speech synthesis model training method of any of the above embodiments.

[0112] The prediction module 52 is also used to encode the feature information through the encoder and attention network of the speech synthesis model to obtain encoded data; input the standard Mel spectrum of the object to be synthesized into the deep learning network, and use the deep learning network to extract the timbre feature matrix of the object to be synthesized; extract audio features based on the object to be synthesized to obtain the audio feature matrix of the sample audio; and make predictions based on the feature information, the timbre feature matrix and the audio feature matrix through the decoder of the speech synthesis model to obtain the predicted feature spectrum synthesized by the speech synthesis model.

[0113] See also Figure 6 , Figure 6 : is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the training method of the speech synthesis model and the steps of the speech synthesis method of any of the above embodiments. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0114] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of the training method embodiment of any of the above-mentioned speech synthesis models. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0115] The above scheme can achieve the similarity of speech synthesis.

[0116] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 701 that can be executed by a processor, and the program instructions 701 are used to implement the training method of the speech synthesis model and the steps of the speech synthesis method in any of the above embodiments.

[0117] The above scheme can achieve the similarity of speech synthesis.

[0118] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for training a speech synthesis model, characterized in that: The training method of the speech synthesis model includes: Acquire sample audio, and acquire feature information of the sample audio, and a standard feature spectrum of each frame of the sample audio; Predicting the characteristic spectrum of the sample audio based on the sample audio and its characteristic information through an initial model to obtain a predicted characteristic spectrum of each frame of the sample audio; Subtracting the standard characteristic spectrum of each frame of the sample audio from the standard characteristic spectrum of the next frame in sequence to obtain a standard characteristic spectrum error, and subtracting the predicted characteristic spectrum of each frame of the sample audio from the predicted characteristic spectrum of the next frame to obtain a predicted characteristic spectrum error; The initial model is trained using the standard feature spectrum error and the predicted feature spectrum error, and the trained model is determined as the speech synthesis model.

2. The method for training a speech synthesis model according to claim 1, characterized in that: The standard characteristic spectrum includes: a standard Mel spectrum and a standard linear spectrum; the predicted characteristic spectrum includes: a predicted Mel spectrum and a predicted linear spectrum; The steps of sequentially performing a subtraction process on the standard characteristic spectrum of each frame of the sample audio and the standard characteristic spectrum of the next frame to obtain a standard characteristic spectrum error, and performing a subtraction process on the predicted characteristic spectrum of each frame of the sample audio and the predicted characteristic spectrum of the next frame to obtain a predicted characteristic spectrum error include: Subtracting the Mel frequency of each frame in the standard Mel spectrum from the Mel frequency of the next frame in turn to obtain a standard Mel spectrum error between the Mel frequencies of adjacent frames; and subtracting the Mel frequency of each frame in the predicted Mel spectrum from the Mel frequency of the next frame in turn to obtain a predicted Mel spectrum error between the Mel frequencies of adjacent frames; and Subtracting the frequency of each frame in the standard linear spectrum from the frequency of the next frame in turn to obtain a standard linear spectrum error between the frequencies of adjacent frames; and subtracting the frequency of each frame in the predicted linear spectrum from the frequency of the next frame in turn to obtain a predicted linear spectrum error between the frequencies of adjacent frames; The step of training the initial model by using the standard characteristic spectrum error and the predicted characteristic spectrum error and determining the trained model as the speech synthesis model comprises: In response to the similarity between the standard Mel spectrum error and the predicted Mel spectrum error and / or the similarity between the standard linear spectrum error and the predicted linear spectrum error meeting a preset requirement, the training is completed and the speech synthesis model is acquired.

3. The method for training a speech synthesis model according to claim 2, characterized in that: The step of predicting the characteristic spectrum of the sample audio based on the sample audio and its characteristic information by using the initial model to obtain the predicted characteristic spectrum of each frame of the sample audio also includes: Performing convolution processing on the predicted mel spectrum through the initial model; Perform residual processing based on the predicted Mel spectrum before and after convolution processing to obtain a Mel residual spectrum; The step of sequentially performing a subtraction process on the standard characteristic spectrum of each frame of the sample audio and the standard characteristic spectrum of the next frame to obtain a standard characteristic spectrum error, and performing a subtraction process on the predicted characteristic spectrum of each frame of the sample audio and the predicted characteristic spectrum of the next frame to obtain a predicted characteristic spectrum error also includes: Subtracting the frequency of each frame in the Mel residual spectrum from the frequency of the next frame in turn to obtain a prediction residual spectrum error between the frequencies of adjacent frames; The step of training the initial model by using the standard characteristic spectrum error and the predicted characteristic spectrum error and determining the trained model as the speech synthesis model further includes: In response to the similarity between the standard Mel spectrum error and the predicted Mel spectrum error and / or the similarity between the standard linear spectrum error and the predicted linear spectrum error and / or the similarity between the predicted residual spectrum error and the standard Mel spectrum error meeting the preset requirements, the speech synthesis model is acquired.

4. The method for training a speech synthesis model according to claim 2, characterized in that: The step of acquiring the sample audio, acquiring the feature information of the sample audio, and the standard feature spectrum of each frame of the sample audio includes: Acquire the sample audio, and acquire the phonemes, rhythm and pitch of the sample audio, and use the phonemes, rhythm and pitch as the feature information; Performing frame windowing and Fourier transformation on the sample audio to obtain a standard linear spectrum of the sample audio; The standard linear spectrum is input into a Mel filter bank for filtering to obtain the standard Mel spectrum.

5. The method for training a speech synthesis model according to claim 4, characterized in that: The training method of the speech synthesis model also includes: Inputting the standard Mel spectrum of the sample audio into a deep learning network, and using the deep learning network to extract a timbre feature matrix of the sample audio; Audio feature extraction is performed based on the sample audio to obtain an audio feature matrix of the sample audio.

6. The method for training a speech synthesis model according to claim 5, characterized in that: The initial model includes an encoder, an attention network, and a decoder that are cascaded to each other; The step of predicting the characteristic spectrum of the sample audio based on the sample audio and its characteristic information by using the initial model to obtain the predicted characteristic spectrum of each frame of the sample audio comprises: Input the phonemes, rhythm and pitch of the sample audio into the encoder and attention network in the initial model for encoding to obtain encoded data; The encoded data, the audio feature matrix and the timbre feature matrix of the sample audio are input into the decoder in the initial model for decoding to obtain the predicted feature spectrum.

7. The method for training a speech synthesis model according to claim 1, characterized in that: The step of training the initial model by using the standard characteristic spectrum error and the predicted characteristic spectrum error and determining the trained model as the speech synthesis model further includes: Determining a mean square error between each frame in the standard characteristic spectrum and a corresponding frame in the predicted characteristic spectrum based on the standard characteristic spectrum and the predicted characteristic spectrum; The initial model is trained using the mean square error between each frame in the standard feature spectrum and the corresponding frame in the predicted feature spectrum, and the trained model is determined as the speech synthesis model.

8. A speech synthesis method, characterized in that: The speech synthesis method comprises: Obtaining feature information of the object to be synthesized and the target object; Inputting feature information of the object to be synthesized and the target object into a speech synthesis model for prediction, and obtaining a predicted feature spectrum synthesized by the speech synthesis model; Outputting synthesized speech based on the predicted feature spectrum by the speech synthesis model; Wherein, the speech synthesis model is trained by the speech synthesis model training method described in any one of claims 1-7.

9. The speech synthesis method according to claim 8, characterized in that: The speech synthesis model includes an encoder, an attention network and a decoder that are cascaded to each other; The step of inputting the feature information of the object to be synthesized and the target object into a speech synthesis model for prediction to obtain a predicted feature spectrum synthesized by the speech synthesis model comprises: Encoding the feature information through the encoder and attention network of the speech synthesis model to obtain encoded data; Inputting the standard Mel spectrum of the object to be synthesized into a deep learning network, and using the deep learning network to extract the timbre feature matrix of the object to be synthesized; Extract audio features based on the object to be synthesized to obtain an audio feature matrix of the sample audio; The decoder of the speech synthesis model performs prediction based on the feature information, the timbre feature matrix and the audio feature matrix to obtain a predicted feature spectrum synthesized by the speech synthesis model.

10. A training device for a speech synthesis model, characterized in that: The training device of the speech synthesis model comprises: An acquisition module, used to acquire sample audio, and acquire feature information of the sample audio, and a standard feature spectrum of each frame of the sample audio; A prediction module, used to predict the characteristic spectrum of the sample audio based on the sample audio and its characteristic information through an initial model to obtain a predicted characteristic spectrum of each frame of the sample audio; A calculation module, used for sequentially performing a difference processing between the standard characteristic spectrum of each frame of the sample audio and the standard characteristic spectrum of the next frame to obtain a standard characteristic spectrum error, and performing a difference processing between the predicted characteristic spectrum of each frame of the sample audio and the predicted characteristic spectrum of the next frame to obtain a predicted characteristic spectrum error; A training module is used to train the initial model using the standard feature spectrum error and the predicted feature spectrum error, and determine the trained model as the speech synthesis model.

11. A speech synthesis device, characterized in that: The speech synthesis device comprises: An acquisition module is used to acquire feature information of the object to be synthesized and the target object; A prediction module, used for inputting feature information of the object to be synthesized and the target object into a speech synthesis model for prediction, and obtaining a predicted feature spectrum synthesized by the speech synthesis model; An output module, configured to output synthesized speech based on the predicted feature spectrum through the speech synthesis model; Wherein, the speech synthesis model is trained by the speech synthesis model training method described in any one of claims 1-7.

12. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the training method of the speech synthesis model as claimed in any one of claims 1 to 7 or the speech synthesis method as claimed in any one of claims 8 to 9.

13. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the training method of the speech synthesis model as claimed in any one of claims 1 to 7 or the speech synthesis method as claimed in any one of claims 8 to 9 is implemented.