Speech synthesis method and model training method thereof, related device, equipment and medium

Abstract

The present application discloses a speech synthesis method and its model training method, related devices, equipment and media. The training method of the speech synthesis model includes: obtaining sample audio and obtaining feature information of the sample audio; obtaining a sample Mel spectrum of the sample audio through a phoneme duration prediction model, obtaining sample phoneme weight information of the sample audio, and obtaining sample phoneme duration; inputting the feature information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model to obtain sample synthesized speech and predicted phoneme weight information; adjusting the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information. The above scheme can learn very accurate phoneme duration information to accelerate the convergence of the model, and can efficiently control the rhythm of the synthesized speech.

Description

Technical Field

[0001] The present application relates to the technical field of speech synthesis, and in particular to a speech synthesis method and a model training method thereof, and related devices, equipment and media. Background Art

[0002] With the continuous development of electronic information processing technology, voice, as an important carrier for people to obtain information, has been widely used in daily life and work. Application scenarios involving voice usually include speech synthesis processing, which refers to synthesizing specified text or audio into target audio that meets the requirements.

[0003] Take singing synthesis as an example. Singing synthesis is an important branch of speech synthesis. Singing synthesis has high requirements for rhythm and is difficult to learn. In addition, since singing synthesis requires more professional recorders (who need to have a certain foundation in vocal theory) and requires clean a cappella data, data acquisition is a big problem. Usually, rhythm information can be synthesized into the input of the speech synthesis model in digital form according to the frame resolution, but a large amount of data is required and the convergence is relatively slow. In addition, during speech synthesis, there will be a certain amount of uncontrollable rhythm. Summary of the invention

[0004] The present application provides a speech synthesis method and a model training method thereof, related devices, equipment and media, which can accelerate the convergence of the model and efficiently control the rhythm of the synthesized speech.

[0005] In a first aspect, the present application provides a method for training a speech synthesis model, which includes: obtaining sample audio and obtaining feature information of the sample audio; obtaining a sample mel spectrum of the sample audio through a phoneme duration prediction model, obtaining sample phoneme weight information of the sample audio, and obtaining sample phoneme duration; inputting the feature information of the sample audio, the sample mel spectrum and the sample phoneme duration into the speech synthesis model to obtain sample synthesized speech and predicted phoneme weight information; and adjusting the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information.

[0006] The acquiring of the sample audio and the acquiring of the characteristic information of the sample audio includes: acquiring the sample audio and acquiring the phonemes and pitches of the sample audio, and using the phonemes and pitches as the characteristic information.

[0007] Among them, the phoneme duration prediction model includes a first attention network; the obtaining of the sample mel spectrum of the sample audio through the phoneme duration prediction model, obtaining the sample phoneme weight information of the sample audio, and obtaining the sample phoneme duration includes: inputting the sample audio into the phoneme duration prediction model, outputting the sample mel spectrum of the sample audio, and obtaining the sample phoneme weight information of the sample audio through the first attention network, and obtaining the sample phoneme duration based on the sample phoneme weight information.

[0008] Among them, the step of obtaining the sample phoneme weight information of the sample audio through the first attention network, and obtaining the sample phoneme duration based on the sample phoneme weight information, includes: obtaining the alignment information of the first attention network, the alignment information is a two-dimensional matrix, the row coordinates of the two-dimensional matrix correspond to the phoneme set of the sample audio, and the column coordinates of the two-dimensional matrix correspond to the audio frame of the sample audio; screening the alignment information, looping the column coordinates, determining the phoneme with the largest weight corresponding to each audio frame of the sample audio, and then obtaining the number of audio frames corresponding to each phoneme; determining the sample phoneme duration based on the number of audio frames corresponding to each phoneme.

[0009] In which, the speech synthesis model includes a cascaded encoder, a second attention network and a decoder; the feature information of the sample audio, the sample Mel spectrum and the sample phoneme duration are input into the speech synthesis model to obtain sample synthesized speech and predicted phoneme weight information, including: encoding the feature information of the sample audio and the sample phoneme duration through the encoder in the speech synthesis model to obtain encoded data; decoding the sample Mel spectrum and the encoded data in turn through the second attention network and the decoder in the speech synthesis model to obtain the sample synthesized speech, and obtaining the predicted phoneme weight information corresponding to the sample audio through the second attention network.

[0010] Among them, adjusting the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information includes: determining the mean square error between the sample phoneme weight information and the predicted phoneme weight information; training the speech synthesis model using the mean square error between the sample phoneme weight information and the predicted phoneme weight information, and determining the trained model as the speech synthesis model.

[0011] A second aspect of the present application provides a speech synthesis method, which includes: acquiring feature information and phoneme duration of a target object; and acquiring an object to be synthesized, and determining a mel spectrum of the object to be synthesized; encoding the feature information and phoneme duration of the target object through a speech synthesis model to obtain encoded data; decoding the mel spectrum of the object to be synthesized and the encoded data through the speech synthesis model to obtain synthesized speech of the object to be synthesized; wherein the speech synthesis model is trained by the training method of the speech synthesis model in the first aspect above.

[0012] The third aspect of the present application provides a training device for a speech synthesis model, the training device for the speech synthesis model comprising: a first acquisition module, the first acquisition module is used to acquire sample audio and obtain feature information of the sample audio; a second acquisition module, the second acquisition module is used to acquire a sample Mel spectrum of the sample audio through a phoneme duration prediction model, obtain sample phoneme weight information of the sample audio, and obtain sample phoneme duration; a prediction module, the prediction module is used to input the feature information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model to obtain sample synthesized speech and predicted phoneme weight information; an optimization module, the optimization module is used to adjust the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information.

[0013] The fourth aspect of the present application provides a speech synthesis device, which includes: an acquisition module, used to acquire feature information and phoneme duration of a target object; and to acquire an object to be synthesized and determine a mel spectrum of the object to be synthesized; an encoding module, used to encode the feature information and phoneme duration of the target object through a speech synthesis model to obtain encoded data; a decoding module, used to decode the mel spectrum of the object to be synthesized and the encoded data through the speech synthesis model to obtain a synthesized speech of the object to be synthesized; wherein the speech synthesis model is trained by the training method of the speech synthesis model in the first aspect above.

[0014] The fifth aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the training method of the speech synthesis model in the first aspect above, or the speech synthesis method in the second aspect above.

[0015] The sixth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the training method for the speech synthesis model in the first aspect or the speech synthesis method in the second aspect.

[0016] In the above scheme, the present application obtains sample audio and characteristic information of the sample audio, and then obtains the sample Mel spectrum of the sample audio through a phoneme duration prediction model, and obtains the sample phoneme weight information of the sample audio, thereby obtaining the sample phoneme duration, and then inputs the characteristic information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model, so as to obtain the sample synthesized speech and the predicted phoneme weight information. Therefore, based on the difference between the sample phoneme weight information and the predicted phoneme weight information, the network parameters of the speech synthesis model can be adjusted, and the speech synthesis model can be trained by utilizing the difference between the sample phoneme weight information and the predicted phoneme weight information. The speech synthesis model can be optimized, and a supervised learning training method is adopted to learn very accurate phoneme duration information, which can accelerate the convergence of the model. The obtained speech synthesis model has high stability and robustness, is friendly to small batch data, and is of great help in controlling the rhythm of synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of an embodiment of a training method for a speech synthesis model of the present application;

[0018] Figure 2 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S13;

[0019] Figure 3 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S14;

[0020] Figure 4 It is a flowchart of an embodiment of the speech synthesis method of the present application;

[0021] Figure 5 It is a schematic diagram of the framework of an embodiment of a training device for speech synthesis of the present application;

[0022] Figure 6 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application;

[0023] Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0024] Figure 8 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0026] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0027] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects. There can be three kinds of relationships. For example, A and / or B can be: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally means that the associated objects before and after are in an "or" relationship. In addition, "more" in this article means two or more than two.

[0028] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a method for training a speech synthesis model of the present application. In this embodiment, the method for training a speech synthesis model may include the following steps:

[0029] Step S11: Acquire sample audio and obtain feature information of the sample audio.

[0030] Specifically, sample audio is obtained, and feature information of the sample audio can be obtained based on the sample audio. The feature information of the sample audio may include one or more of the feature information of the sample audio, such as phonemes, pitch, and rhythm. Phonemes are the smallest units of speech divided according to the natural properties of speech. Pitch refers to various sounds of different pitches, that is, the height of the sound, which is one of the basic features of the sound. Rhythm includes the level and oblique format and rhyme rules of the sample audio.

[0031] In a specific application scenario, feature information of the sample audio can be obtained based on the sample audio through manual or pre-trained deep learning models.

[0032] In one embodiment, the above step S11 may specifically include: obtaining the sample audio, and obtaining the phonemes and pitches of the sample audio, and using the phonemes and pitches as the feature information. Specifically, the sample audio may be recorded by multiple recorders with different timbres based on the required phonemes and pitches, wherein the sample audio may cover most or all of the pitch range and the phoneme duration range; and then all the phonemes and pitches of the sample audio are extracted from the sample audio. It can be understood that using the phonemes and pitches of the sample audio as the basis for speech synthesis can enrich the features of the sample audio and improve the feature accuracy of the sample audio, thereby enabling speech synthesis to be more complete and accurate based on richer features, thereby improving the effect of speech synthesis.

[0033] In a specific application scenario, when obtaining the phonemes and pitch of the sample audio, the sample audio can be input into a pre-trained deep learning model for prediction to obtain the phonemes and pitch of the sample audio. In another specific application scenario, when obtaining the phonemes and pitch of the sample audio, the phonemes of the sample audio can be analyzed based on the dictionary corresponding to the sample audio, and the pitch of the sample audio can be analyzed based on the international standard sound. In another specific application scenario, when obtaining the phonemes and pitch of the sample audio, the sample audio can be manually analyzed to obtain the phonemes and pitch of the sample audio.

[0034] Step S12: obtaining a sample mel spectrum of the sample audio through a phoneme duration prediction model, obtaining sample phoneme weight information of the sample audio, and obtaining a sample phoneme duration.

[0035] The phoneme duration is the pronunciation duration of the corresponding phoneme. In speech synthesis, especially in the synthesis of singing or music, since singing or music has rhythm, it is necessary to have accuracy in the prediction of phoneme duration so that each note can be on the beat, that is, accurate prediction of phoneme duration can be used to generate synthesized speech that is easier to match the rhythm. Considering that the cost of manual annotation is too high, this application can build a phoneme duration prediction model to predict the phoneme duration. The sample Mel spectrum of the sample audio can be obtained through the phoneme duration prediction model, and the sample phoneme weight information of the sample audio can be obtained at the same time; the sample phoneme weight information records the phoneme weight proportion corresponding to each audio frame of the sample audio. For example, when synthesizing the sentence "I and you", when synthesizing the sound of "I", the coding information used mainly comes from the word "I", and has little to do with the following words. At this time, the weight of the word "I" is set to 1, and the weight of the two words "and you" is set to 0, so the phoneme duration of each phoneme in each audio frame can be obtained, and then the sample phoneme duration corresponding to the sample audio can be obtained.

[0036] Further, the phoneme duration prediction model includes a first attention network. The above step S12 may specifically include: inputting the sample audio into the phoneme duration prediction model, outputting the sample Mel spectrum of the sample audio, and obtaining the sample phoneme weight information of the sample audio through the first attention network, and obtaining the sample phoneme duration based on the sample phoneme weight information. Specifically, the phoneme duration prediction model may be composed of three parts, the first part is an encoding module, the encoding module uses an embedding layer (embedding) + a bidirectional gated recurrent unit (GRU) network; the second part is a decoding module, the decoding module uses a prenet layer + a bidirectional long short-term memory (Long Short-Term Memory, LSTM) network; the encoding module and the decoding module are connected with a first attention network. By inputting the sample audio into the encoding module of the phoneme duration prediction model, the decoding module outputs the sample Mel spectrum of the sample audio, and the sample phoneme weight information of the sample audio can be obtained through the first attention network, and then the sample phoneme duration is obtained based on the sample phoneme weight information.

[0037] Furthermore, the above-mentioned step of obtaining the sample phoneme weight information of the sample audio through the first attention network, and obtaining the sample phoneme duration based on the sample phoneme weight information, may specifically include: obtaining the alignment information of the first attention network, the alignment information is a two-dimensional matrix, the row coordinates of the two-dimensional matrix correspond to the phoneme set of the sample audio, and the column coordinates of the two-dimensional matrix correspond to the audio frame of the sample audio; screening the alignment information, looping the column coordinates, determining the phoneme with the largest weight corresponding to each audio frame of the sample audio, and then obtaining the number of audio frames corresponding to each phoneme; determining the sample phoneme duration based on the number of audio frames corresponding to each phoneme.

[0038] Specifically, the sample audio is trained with the constructed phoneme duration prediction model. When the phoneme duration prediction model converges, the alignment information corresponding to the first attention network is saved. The alignment information is a two-dimensional matrix. The row coordinates of the two-dimensional matrix correspond to the phoneme set of the sample audio, and the column coordinates of the two-dimensional matrix correspond to the audio frame of the sample audio. Therefore, the alignment information can be screened by rules, and the column coordinates can be looped to find the phoneme with the largest weight in each audio frame of the sample audio. Finally, the number of audio frames corresponding to each phoneme can be obtained. According to the duration of each audio frame and the number of audio frames corresponding to a certain phoneme, the phoneme duration corresponding to the phoneme can be obtained.

[0039] Furthermore, the first attention network adopts a forward location-based attention mechanism (Forward LocationBased Attention), which takes into account the forward regularity of the predicted alignment information, can accelerate the convergence of the model, reduce invalid analysis and consumption, and improve model efficiency.

[0040] Step S13: inputting the feature information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model to obtain sample synthesized speech and predicted phoneme weight information.

[0041] Step S14: adjusting the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information.

[0042] Mel spectrum, also known as mel spectrogram, can reflect audio features. After obtaining the feature information of the sample audio, the sample mel spectrum and the sample phoneme duration, the sample audio and its feature information, the sample mel spectrum and the sample phoneme duration can be input into the speech synthesis model, and the speech synthesis model outputs the synthesized speech based on the sample audio and its feature information, the sample mel spectrum and the sample phoneme duration.

[0043] The attention mechanism is also used in the speech synthesis model, and the attention mechanism can learn the corresponding predicted phoneme weight information. Therefore, the sample phoneme weight information obtained by the phoneme duration prediction model is used as the target value, and the predicted phoneme weight information obtained by the speech synthesis model is used as the predicted value. By comparing the sample phoneme weight information and the predicted phoneme weight information, the network parameters of the speech synthesis model are adjusted based on the difference between the sample phoneme weight information and the predicted phoneme weight information to update the speech synthesis model; that is, after obtaining the difference between the sample phoneme weight information and the predicted phoneme weight information, the speech synthesis model is trained using the difference between the sample phoneme weight information and the predicted phoneme weight information, and the trained model is determined as the speech synthesis model.

[0044] In addition, during the training process of the speech synthesis model, the convergence of the difference between the sample phoneme weight information and the predicted phoneme weight information can be obtained. When the difference between the sample phoneme weight information and the predicted phoneme weight information converges, the updating of the network parameters of the speech synthesis model can be stopped. When the difference between the sample phoneme weight information and the predicted phoneme weight information does not converge, the number of adjustments of the network parameters can be obtained. When the number of adjustments reaches a preset number, the final speech synthesis model can be determined based on the network parameters at this time to avoid the loss function not converging and affecting the training efficiency.

[0045] In a specific application scenario, the speech synthesis model can be trained based on the difference between the sample phoneme weight information and the predicted phoneme weight information through a loss function until the loss function converges to obtain a trained speech synthesis model. The loss function may include a cross entropy loss function, an absolute value loss function, a square loss function, etc., which are not specifically limited here.

[0046] In one embodiment, the speech synthesis model includes an encoder, a second attention network, and a decoder that are cascaded to each other. Figure 2 , Figure 2 yes Figure 1 FIG. 1 is a flow chart of an embodiment of step S13 in FIG. 1 ; in this embodiment, the above step S13 may specifically include:

[0047] Step S131: Encode the characteristic information of the sample audio and the sample phoneme duration through the encoder in the speech synthesis model to obtain encoded data.

[0048] Step S132: The sample mel spectrum and the encoded data are decoded in sequence through the second attention network and the decoder in the speech synthesis model to obtain the sample synthesized speech, and the predicted phoneme weight information corresponding to the sample audio is obtained through the second attention network.

[0049] After obtaining the characteristic information of the sample audio and the sample phoneme duration, the characteristic information of the sample audio and the sample phoneme duration are input into the encoder in the speech synthesis model for encoding to obtain encoded data.

[0050] The speech synthesis model of this embodiment is based on the tacotron1 model and improved. Specifically, the speech synthesis model of this embodiment includes a mutually cascaded encoder, a second attention network, and a decoder. Among them, since the feature extraction of phonemes and pitches does not need to take into account contextual information, this embodiment removes the bidirectional GRU module in the tacotron1 model to simplify the speech synthesis model structure, reduce invalid model operations, and thus improve the speech synthesis efficiency of the speech synthesis model and the final speech synthesis model.

[0051] Furthermore, this embodiment replaces the Bahdanau Attention mechanism in the original tacotron1 model with a Forward Location Based Attention mechanism, that is, the speech synthesis model adopts a forward location-based attention mechanism, wherein the forward location-based attention mechanism ignores or reduces silence in speech data, thereby reducing ineffective analysis and consumption of silence in speech synthesis, thereby improving the efficiency and quality of speech synthesis.

[0052] In a specific application scenario, the speech synthesis model can also add a Stop Token network to segment the decoded data through the StopToken network, thereby separating the entire segment of synthesized speech to make it conform to the pronunciation rules, thereby improving the quality of speech synthesis.

[0053] In a specific application scenario, the input of the speech synthesis model can be (iou C3 10), where iou is the phoneme information, C3 is the pitch information, and 10 is the phoneme duration corresponding to the phoneme information.

[0054] In this embodiment, the phonemes, pitches and duration of the sample audio are input into the encoder in the speech synthesis model for encoding, and the output of the encoder is used as the encoded data. After obtaining the encoded data, the sample Mel spectrum and the encoded data are input into the second attention network for processing, and then the output of the second attention network is input into the decoder in the speech synthesis model for decoding. After the decoding is completed, the decoder of the speech synthesis model outputs the sample synthesized speech.

[0055] Please combine Figure 3 , Figure 3 yes Figure 1 FIG. 1 is a flow chart of an embodiment of step S14. In this embodiment, the above step S14 may specifically include:

[0056] Step S141: Determine the mean square error between the sample phoneme weight information and the predicted phoneme weight information.

[0057] Step S142: training the speech synthesis model using the mean square error between the sample phoneme weight information and the predicted phoneme weight information, and determining the trained model as the speech synthesis model.

[0058] When training the speech synthesis model through the difference between the sample phoneme weight information and the predicted phoneme weight information, the mean square error between the sample phoneme weight information and the predicted phoneme weight information can be calculated, and then the speech synthesis model can be trained using the mean square error between the sample phoneme weight information and the predicted phoneme weight information.

[0059] Through the above scheme, the training method of the speech synthesis model of this embodiment obtains sample audio and characteristic information of the sample audio, then obtains the sample Mel spectrum of the sample audio through the phoneme duration prediction model, and obtains the sample phoneme weight information of the sample audio, and then obtains the sample phoneme duration, and then inputs the characteristic information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model, and the sample synthesized speech and the predicted phoneme weight information can be obtained. Therefore, based on the difference between the sample phoneme weight information and the predicted phoneme weight information, the network parameters of the speech synthesis model can be adjusted to realize the training of the speech synthesis model by using the difference between the sample phoneme weight information and the predicted phoneme weight information, and the speech synthesis model can be optimized. By adopting the supervised learning training method, very accurate phoneme duration information can be learned, and the convergence of the model can be accelerated. The obtained speech synthesis model has high stability and robustness, is friendly to small batch data, and is of great help in controlling the rhythm of synthesized speech.

[0060] See also Figure 4 , Figure 4 1 is a flow chart of an embodiment of a speech synthesis method of the present application. In this embodiment, the speech synthesis method may include the following steps:

[0061] Step S41: Acquire the characteristic information and phoneme duration of the target object.

[0062] First, the characteristic information and phoneme duration of the target object are obtained. The characteristic information of the target object includes phonemes and pitches. The target object may include target audio such as songs, operas, broadcasts or recitations. The specific type of the target object is not limited here. Phonemes, pitches and phoneme durations can reflect the speech characteristics of the target object as much as possible.

[0063] In a specific application scenario, when obtaining the phonemes, pitches, and phoneme duration of a target object, the target object can be input into a pre-trained deep learning model for prediction to obtain the phonemes, pitches, and phoneme duration of the target object. In another specific application scenario, when obtaining the phonemes, pitches, and phoneme duration of a target object, the phonemes of the target object can be analyzed based on a dictionary corresponding to the target object, the pitch of the target object can be analyzed based on international standard sounds, and the target object can be predicted through a pre-trained deep learning model to obtain the phoneme duration of the target object.

[0064] Among them, the speech synthesis of this embodiment needs to improve the effect of speech synthesis. Therefore, the phonemes, pitches and phoneme durations of the target object obtained in this step are indispensable, so as to refine the speech features of the target object to the greatest extent, so that when performing subsequent speech synthesis, the feature accuracy of the synthesized speech can be improved based on the multiple features of the phonemes, pitches and phoneme durations of the target object, thereby improving the accuracy of the synthesized speech.

[0065] Step S42: Acquire the object to be synthesized, and determine the mel spectrum of the object to be synthesized.

[0066] The object to be synthesized in this embodiment may include audio to be synthesized or other voice data containing pronunciation data. Among them, the speech synthesis of this embodiment can be applied to scenes such as audio production of music software and audio tuning of karaoke software. For example: the object to be synthesized can be voice audio input by the user, or it can be music scores, opera scores or other arbitrary texts containing pronunciation data made by the user. The audio to be synthesized may also include audio data emitted artificially or audio data output by a deep neural network, wherein the specific type of the object to be synthesized can be set based on the actual situation and is not limited here.

[0067] After the object to be synthesized is obtained, the Mel spectrum of the object to be synthesized is determined based on the object to be synthesized. In a specific application scenario, the Mel spectrum can be obtained by first obtaining the linear spectrum of the object to be synthesized, and then performing a weighted summation of the linear spectrum on a Mel scale. In another specific application scenario, the Mel spectrum can be obtained by first obtaining the linear spectrum of the object to be synthesized, and then inputting the linear spectrum into a Mel filter group for filtering. In yet another specific application scenario, the Mel spectrum of the object to be synthesized can be obtained by obtaining feature information of the object to be synthesized, and then inputting the feature information of the object to be synthesized into a phoneme duration prediction model. The specific method for obtaining the Mel spectrum is not limited here.

[0068] The execution order of this step and step S41 is not specific, and step S41 can be executed first and then step S42, or step S42 can be executed first and then step S41, or step S42 and step S41 can be executed simultaneously. Specifically, step S41 only needs to be executed before executing step S43, and step S42 only needs to be executed before executing step S44.

[0069] Step S43: Encode the characteristic information and phoneme duration of the target object through a speech synthesis model to obtain encoded data.

[0070] The speech synthesis model of this embodiment is a speech synthesis model trained by the training method of the speech synthesis model of any of the above embodiments.

[0071] After the phonemes, pitches and phoneme durations of the target object are obtained, the phonemes, pitches and phoneme durations of the target object are encoded through a speech synthesis model to obtain encoded data.

[0072] In this step of other embodiments, the phonemes, pitches and phoneme durations of the target object can also be encoded by an independent encoder to obtain encoded data. Among them, the encoder can include one or more encoders such as AutoEncoder, Sparse AutoEncoder, Denoising AutoEncoders, waveform encoders, parameter encoders, hybrid encoders, etc. In another specific application scenario, the phonemes, pitches and phoneme durations of the target object can also be encoded by encoding methods such as hard coding, unique hot coding or target variable coding to obtain encoded data.

[0073] Step S44: decoding the mel-spectrogram of the object to be synthesized and the encoded data through the speech synthesis model to obtain the synthesized speech of the object to be synthesized.

[0074] After obtaining the mel spectrum and encoding data of the object to be synthesized, the speech synthesis model outputs the synthesized speech based on the mel spectrum and encoding data of the object to be synthesized, thereby completing the speech synthesis of the object to be synthesized.

[0075] Through the above steps, after obtaining the Mel-spectrogram of the object to be synthesized and the characteristic information and phoneme duration of the target object, the speech synthesis method of this embodiment inputs the characteristic information and phoneme duration of the target object and the Mel-spectrogram of the object to be synthesized into the speech synthesis model, and outputs the synthesized speech of the object to be synthesized. Since the speech synthesis model of this embodiment is a speech synthesis model trained by the training method of the speech synthesis model of any of the above embodiments, it is trained based on the difference between the sample phoneme weight information and the predicted phoneme weight information, and the speech synthesis model can be optimized. By adopting the supervised learning training method, very accurate phoneme duration information can be learned, which can accelerate the convergence of the model. The obtained speech synthesis model has high stability and robustness, is friendly to small batch data, and is of great help in controlling the rhythm of synthesized speech.

[0076] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of the training device for speech synthesis of the present application. The training device 50 for the speech synthesis model includes a first acquisition module 500, a second acquisition module 502, a prediction module 504 and an optimization module 506. The first acquisition module 500 is used to obtain the sample audio and obtain the feature information of the sample audio; the second acquisition module 502 is used to obtain the sample Mel spectrum of the sample audio through the phoneme duration prediction model, obtain the sample phoneme weight information of the sample audio, and obtain the sample phoneme duration; the prediction module 504 is used to input the feature information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model to obtain the sample synthesized speech and the predicted phoneme weight information; the optimization module 506 is used to adjust the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information.

[0077] In the above scheme, the sample audio is acquired through the first acquisition module 500, and the feature information of the sample audio is acquired. Then the second acquisition module 502 acquires the sample Mel spectrum of the sample audio through the phoneme duration prediction model, and acquires the sample phoneme weight information of the sample audio, and then obtains the sample phoneme duration. Then the prediction module 504 inputs the feature information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model, and the sample synthesized speech and the predicted phoneme weight information can be obtained. Therefore, the optimization module 506 can adjust the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information, and realize the training of the speech synthesis model by using the difference between the sample phoneme weight information and the predicted phoneme weight information. The speech synthesis model can be optimized, and the supervised learning training method is adopted to learn very accurate phoneme duration information, which can accelerate the convergence of the model. The obtained speech synthesis model has high stability and robustness, is friendly to small batch data, and is very helpful for controlling the rhythm of synthesized speech.

[0078] In some embodiments, the first acquisition module 500 executes the steps of acquiring sample audio and acquiring feature information of the sample audio, including: acquiring the sample audio, acquiring the phonemes and pitches of the sample audio, and using the phonemes and pitches as the feature information.

[0079] In some embodiments, the phoneme duration prediction model includes a first attention network; the second acquisition module 502 executes the steps of acquiring a sample mel-spectrogram of the sample audio through the phoneme duration prediction model, acquiring sample phoneme weight information of the sample audio, and obtaining the sample phoneme duration, including: inputting the sample audio into the phoneme duration prediction model, outputting the sample mel-spectrogram of the sample audio, and acquiring the sample phoneme weight information of the sample audio through the first attention network, and obtaining the sample phoneme duration based on the sample phoneme weight information.

[0080] In some embodiments, the second acquisition module 502 executes the steps of acquiring the sample phoneme weight information of the sample audio through the first attention network, and obtaining the sample phoneme duration based on the sample phoneme weight information, specifically including: obtaining the alignment information of the first attention network, the alignment information is a two-dimensional matrix, the row coordinates of the two-dimensional matrix correspond to the phoneme set of the sample audio, and the column coordinates of the two-dimensional matrix correspond to the audio frame of the sample audio; filtering the alignment information, looping the column coordinates, determining the phoneme with the largest weight corresponding to each audio frame of the sample audio, and then obtaining the number of audio frames corresponding to each phoneme; determining the sample phoneme duration based on the number of audio frames corresponding to each phoneme.

[0081] In some embodiments, the speech synthesis model includes a cascaded encoder, a second attention network and a decoder; the prediction module 504 executes the step of inputting the feature information of the sample audio, the sample mel spectrum and the sample phoneme duration into the speech synthesis model to obtain sample synthesized speech and predicted phoneme weight information, including: encoding the feature information of the sample audio and the sample phoneme duration through the encoder in the speech synthesis model to obtain encoded data; decoding the sample mel spectrum and the encoded data in turn through the second attention network and the decoder in the speech synthesis model to obtain the sample synthesized speech, and obtaining the predicted phoneme weight information corresponding to the sample audio through the second attention network.

[0082] In some embodiments, the optimization module 506 performs the step of adjusting the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information, including: determining the mean square error between the sample phoneme weight information and the predicted phoneme weight information; training the speech synthesis model using the mean square error between the sample phoneme weight information and the predicted phoneme weight information, and determining the trained model as the speech synthesis model.

[0083] See also Figure 6 , Figure 6It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application. The speech synthesis device 60 includes: an acquisition module 600, which is used to acquire the characteristic information and phoneme duration of the target object, and acquire the object to be synthesized, and determine the Mel spectrum of the object to be synthesized; an encoding module 602, which is used to encode the characteristic information and phoneme duration of the target object through a speech synthesis model to obtain encoded data; a decoding module 604, which is used to decode the Mel spectrum of the object to be synthesized and the encoded data through the speech synthesis model to obtain the synthesized speech of the object to be synthesized; wherein the speech synthesis model is trained by the training method of the speech synthesis model of any of the above embodiments.

[0084] See also Figure 7 , Figure 7 : is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 70 includes a memory 701 and a processor 702 coupled to each other, and the processor 702 is used to execute program instructions stored in the memory 701 to implement the steps of any of the above-mentioned speech synthesis model training method embodiments, or the steps of any of the above-mentioned speech synthesis method embodiments. In a specific implementation scenario, the electronic device 70 may include but is not limited to: a microcomputer, a server.

[0085] Specifically, the processor 702 is used to control itself and the memory 701 to implement the steps of the training method embodiment of any of the above-mentioned speech synthesis models, or the steps of any of the above-mentioned speech synthesis method embodiments. The processor 702 can also be called a CPU (Central Processing Unit). The processor 702 may be an integrated circuit chip with signal processing capabilities. The processor 702 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 702 can be implemented by an integrated circuit chip.

[0086] See also Figure 8 , Figure 8The computer-readable storage medium 80 stores program instructions 800 that can be executed by a processor, and the program instructions 800 are used to implement the steps of any of the above-mentioned speech synthesis model training method embodiments, or the steps of any of the above-mentioned speech synthesis method embodiments.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for training a speech synthesis model, characterized in that: The speech synthesis model includes an encoder, a second attention network, and a decoder that are cascaded to each other. The training method of the speech synthesis model includes: Acquire sample audio and obtain feature information of the sample audio; A sample mel spectrum of the sample audio is obtained through a phoneme duration prediction model, sample phoneme weight information of the sample audio is obtained, and the sample phoneme duration is obtained; the phoneme duration prediction model includes a first attention network, wherein the sample audio is input into the phoneme duration prediction model, the sample mel spectrum of the sample audio is output, and the sample phoneme weight information of the sample audio is obtained through the first attention network, and the sample phoneme duration is obtained based on the sample phoneme weight information; Input the characteristic information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model to obtain sample synthesized speech and predicted phoneme weight information; wherein, the characteristic information of the sample audio and the sample phoneme duration are encoded by the encoder in the speech synthesis model to obtain encoded data; the sample Mel spectrum and the encoded data are decoded in turn by the second attention network and the decoder in the speech synthesis model to obtain the sample synthesized speech, and the predicted phoneme weight information corresponding to the sample audio is obtained by the second attention network; Based on the difference between the sample phoneme weight information and the predicted phoneme weight information, the network parameters of the speech synthesis model are adjusted.

2. The method for training a speech synthesis model according to claim 1, characterized in that: The step of obtaining the sample audio and acquiring feature information of the sample audio includes: The sample audio is obtained, and the phonemes and pitches of the sample audio are obtained, and the phonemes and pitches are used as the feature information.

3. The method for training a speech synthesis model according to claim 1, characterized in that: The step of acquiring the sample phoneme weight information of the sample audio through the first attention network, and obtaining the sample phoneme duration based on the sample phoneme weight information includes: Acquire alignment information of the first attention network, where the alignment information is a two-dimensional matrix, where row coordinates of the two-dimensional matrix correspond to a phoneme set of the sample audio, and column coordinates of the two-dimensional matrix correspond to an audio frame of the sample audio; The alignment information is screened, column coordinates are looped, and the phoneme with the largest weight corresponding to each audio frame of the sample audio is determined, thereby obtaining the number of audio frames corresponding to each phoneme; The sample phoneme duration is determined based on the number of audio frames corresponding to each phoneme.

4. The method for training a speech synthesis model according to claim 1, characterized in that: The adjusting the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information includes: Determining a mean square error between the sample phoneme weight information and the predicted phoneme weight information; The speech synthesis model is trained using the mean square error between the sample phoneme weight information and the predicted phoneme weight information, and the trained model is determined as the speech synthesis model.

5. A speech synthesis method, characterized in that: The speech synthesis method comprises: Obtaining feature information and phoneme duration of the target object; and Acquire the object to be synthesized, and determine the mel spectrum of the object to be synthesized; Encoding the characteristic information and phoneme duration of the target object through a speech synthesis model to obtain encoded data; Decoding the mel spectrum of the object to be synthesized and the encoded data through the speech synthesis model to obtain the synthesized speech of the object to be synthesized; Wherein, the speech synthesis model is trained by the speech synthesis model training method described in any one of claims 1-4.

6. A training device for a speech synthesis model, characterized in that: The speech synthesis model includes an encoder, a second attention network and a decoder which are cascaded to each other. The training device of the speech synthesis model includes: A first acquisition module, the first acquisition module is used to acquire a sample audio and acquire feature information of the sample audio; A second acquisition module, the second acquisition module is used to obtain a sample Mel spectrum of the sample audio through a phoneme duration prediction model, obtain sample phoneme weight information of the sample audio, and obtain the sample phoneme duration; the phoneme duration prediction model includes a first attention network, wherein the sample audio is input into the phoneme duration prediction model, the sample Mel spectrum of the sample audio is output, and the sample phoneme weight information of the sample audio is obtained through the first attention network, and the sample phoneme duration is obtained based on the sample phoneme weight information; A prediction module, the prediction module is used to input the feature information of the sample audio, the sample Mel spectrum and the sample phoneme duration into the speech synthesis model to obtain sample synthesized speech and predicted phoneme weight information; wherein, the feature information of the sample audio and the sample phoneme duration are encoded by the encoder in the speech synthesis model to obtain encoded data; the sample Mel spectrum and the encoded data are decoded in turn by the second attention network and the decoder in the speech synthesis model to obtain the sample synthesized speech, and the predicted phoneme weight information corresponding to the sample audio is obtained by the second attention network; An optimization module, wherein the optimization module is used to adjust the network parameters of the speech synthesis model based on the difference between the sample phoneme weight information and the predicted phoneme weight information.

7. A speech synthesis device, characterized in that: The speech synthesis device comprises: An acquisition module is used to acquire characteristic information and phoneme duration of a target object; and to acquire an object to be synthesized and determine a mel spectrum of the object to be synthesized; An encoding module, used to encode the characteristic information and phoneme duration of the target object through a speech synthesis model to obtain encoded data; A decoding module, used for decoding the mel spectrum of the object to be synthesized and the encoded data through the speech synthesis model to obtain the synthesized speech of the object to be synthesized; Wherein, the speech synthesis model is trained by the speech synthesis model training method described in any one of claims 1-4.

8. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the training method of the speech synthesis model as claimed in any one of claims 1 to 4 or the speech synthesis method as claimed in claim 5.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the training method of the speech synthesis model as claimed in any one of claims 1 to 4 or the speech synthesis method as claimed in claim 5 is implemented.

Images