Speaker pitch prediction method, device, electronic device, and storage medium

Abstract

The present invention discloses a speaker pitch prediction method, device, electronic device, and storage medium. The method comprises: normalizing each pitch of each speaker to obtain an individual normalized pitch of each pitch, inputting multiple individual normalized pitches into an encoder of a text-to-speech model for encoding training; performing global normalization on each individual normalized pitch, inputting the absolute normalized pitch into a decoder of the text-to-speech model for decoding audio feature training; using the trained encoder to predict the pitch of the text to be converted of the speaker to be predicted, and synthesizing the audio features of the text to be converted based on the pitch by the trained encoder and decoder. The present invention performs normalization according to different speakers respectively, ensuring that the pitch distribution of different speakers is the same, thereby improving the accuracy of the encoder in predicting the pitch.

Description

Technical Field

[0001] The present invention relates to the field of speech-related technologies, and in particular to a method, device, electronic device, and storage medium for predicting a speaker's pitch. Background Art

[0002] Text to Speech (TTS) is a speech synthesis technology used to convert text into speech. In the existing TTS technology, multiple speakers are usually trained into a TTS model. The model uses an encoder (encoder) to encode the input text into high-dimensional features, and decodes the high-dimensional features in the encoder through a decoder (decoder) to generate audio features. Pitch features are added during training to stabilize the generated pitch. During prediction, the encoder also needs to predict the pitch and add it to the decoder. Among them, pitch refers to various sounds with different pitches, that is, the height of the sound, which is one of the basic characteristics of the sound. The essence of sound is a mechanical wave, so when the speed of sound is constant, the height of the sound is determined by the frequency and wavelength of the mechanical wave. If the frequency is high and the wavelength is short, the sound is "high", and vice versa, if the frequency is low and the wavelength is long, the sound is "low".

[0003] In existing TTS technology, pitch is usually used as a feature input to the encoder to maintain the accuracy of the synthesized audio pitch. However, the pitch range of different speakers is usually different, and the difference between men and women is even greater, so it needs to be normalized before being added to the model. Normalization is a way to simplify calculations, that is, to transform a dimensional expression into a dimensionless expression and become a scalar.

[0004] However, if all speakers are normalized uniformly, the distribution of each speaker will vary greatly, which will cause inaccurate predictions when the encoder predicts the pitch, resulting in inaccurate pitch of the synthesized audio. Figure 1 FIG. 1 is a schematic diagram of the overall normalized pitch predicted by the encoder of the prior art, wherein the horizontal axis is the audio frame and the vertical axis is the overall normalized pitch. Figure 1 It can be seen that most of the normalized pitch values ​​of the prior art are below the mean value of 0, are unevenly distributed, and have some abnormal points (abnormally high pitch). Summary of the invention

[0005] Based on this, it is necessary to provide a speaker pitch prediction method, device, electronic device, and storage medium to address the technical problem of inaccurate pitch of synthesized audio in the prior art.

[0006] The present invention provides a method for predicting a speaker's pitch, comprising:

[0007] Acquire multiple pitches of each speaker, normalize each pitch of each speaker to obtain an individual normalized pitch of each pitch, and input the multiple individual normalized pitches into an encoder of a text-to-speech model for encoding training;

[0008] Performing global normalization on each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and inputting the absolute normalized pitches into a decoder of a text-to-speech model for decoding audio feature training;

[0009] The trained encoder is used to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder are used to synthesize the audio features of the text to be converted based on the pitch.

[0010] Furthermore, the step of normalizing the pitch of each speaker to obtain an individual normalized pitch includes:

[0011] Calculate the mean pitch and pitch variance of each speaker respectively;

[0012] Based on the pitch mean and pitch variance of each speaker, the individual normalized pitch of each pitch of each speaker is calculated.

[0013] Furthermore, the calculating of the individual normalized pitch of each pitch of each speaker based on the pitch mean and pitch variance of each speaker includes:

[0014] Calculate the individual normalized pitch corresponding to the jth pitch of the i-th speaker as:

[0015] pitch normi,j =(pitch i,j -mean i ) / std i ;

[0016] Among them, pitch normi,j is the individual normalized pitch corresponding to the jth pitch of the ith speaker, pitch i,j is the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker.

[0017] Furthermore, the globally normalizing each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches includes:

[0018] Calculate the total mean of the pitch of all speakers as the global pitch mean, and calculate the total variance of the pitch of all speakers as the global pitch variance;

[0019] Each of the individual normalized pitches is denormalized, and then globally normalized based on the global pitch mean and the global pitch variance to obtain an absolute normalized pitch.

[0020] Furthermore, the denormalizing of each individual normalized pitch and then performing global normalization based on the global pitch mean and the global pitch variance to obtain an absolute normalized pitch includes:

[0021] In one embodiment, the denormalizing each of the individual normalized pitches and then performing global normalization based on a global pitch mean and a global pitch variance to obtain an absolute normalized pitch comprises:

[0022] Calculate the absolute normalized pitch corresponding to the jth pitch of the i-th speaker as:

[0023] pitch fi,i =((pitch normi,j *std i +mean i )-mean) / std;

[0024] Among them, pitch fi is the absolute normalized pitch corresponding to the jth pitch of the ith speaker, pitch normi is the individual normalized pitch corresponding to the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker, mean is the global pitch mean, and std is the global pitch variance.

[0025] Furthermore, the step of using the trained encoder to predict the pitch of the text to be converted of the speaker to be predicted includes:

[0026] Using the trained encoder, based on the text to be converted of the speaker to be predicted, predict the individual normalized pitch of the speaker to be predicted;

[0027] The pitch mean and pitch variance of the speaker to be predicted are obtained, and based on the pitch mean and pitch variance of the speaker to be predicted, an absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated, and the absolute normalized pitch of the speaker to be predicted is used as the pitch of the text to be converted of the speaker to be predicted.

[0028] Furthermore, the step of calculating the absolute normalized pitch of the individual normalized pitch of the speaker to be predicted based on the pitch mean and pitch variance of the speaker to be predicted includes:

[0029] Finding the pitch mean of the speaker to be predicted, the pitch variance of the speaker to be predicted, the global pitch mean and the global pitch variance;

[0030] The absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated as:

[0031] pitch p =((pitch normp *std p +mean p 0-mean) / std;

[0032] Among them, pitch p is the absolute normalized pitch of the speaker to be predicted, pitch normp is the individual normalized pitch of the speaker to be predicted predicted by the trained decoder, mean p is the mean pitch of the speaker to be predicted, std p is the pitch variance of the speaker to be predicted, mean is the global pitch mean, and std is the global pitch variance.

[0033] The present invention provides a speaker pitch prediction device, comprising:

[0034] An encoder training module, used to obtain multiple pitches of each speaker, normalize each pitch of each speaker to obtain an individual normalized pitch of each pitch, and input the multiple individual normalized pitches into an encoder of a text-to-speech model for encoding training;

[0035] A decoder training module, used to perform global normalization on each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and input the absolute normalized pitches into a decoder of a text-to-speech model for decoding audio feature training;

[0036] The prediction module is used to use the trained encoder to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder are used to synthesize the audio features of the text to be converted based on the pitch.

[0037] The present invention provides an electronic device, comprising:

[0038] at least one processor; and,

[0039] a memory communicatively connected to at least one of the processors; wherein,

[0040] The memory stores instructions that can be executed by at least one of the processors. The instructions are executed by at least one of the processors to enable the at least one of the processors to perform the speaker pitch prediction method as described above.

[0041] The present invention provides a storage medium, wherein the storage medium stores computer instructions, and when a computer executes the computer instructions, it is used to execute all steps of the speaker pitch prediction method as described above.

[0042] In the training stage, the present invention normalizes the pitch of each speaker to obtain an individual normalized pitch, and inputs a plurality of the individual normalized pitches into the encoder of the text-to-speech model for encoding training; globally normalizes each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and inputs the absolute normalized pitch into the decoder of the text-to-speech model for decoding audio feature training; in the prediction stage, the trained encoder is used to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder synthesize the audio features of the text to be converted based on the pitch. The present invention performs normalization according to different speakers, ensures that the pitch distribution of different speakers is the same, thereby improving the accuracy of the encoder in predicting pitch. At the same time, the decoder performs back-normalization and global normalization to obtain the absolute normalized pitch, ensuring that the decoder can synthesize the audio features with accurate pitch when decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the overall normalized pitch predicted by the encoder of the prior art;

[0044] Figure 2 This is a working flow chart of a method for predicting a speaker's pitch according to an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of a method for predicting a speaker's pitch according to another embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the best embodiment of the present invention;

[0047] Figure 5 A schematic diagram of individual normalized pitch predicted by an encoder in one example of the present invention;

[0048] Figure 6 A schematic diagram of a speaker pitch prediction device according to an embodiment of the present invention;

[0049] Figure 7 The figure is a schematic diagram of the hardware structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0050] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. The same components are represented by the same reference numerals. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0051] like Figure 2 FIG. 1 is a flowchart of a method for predicting a speaker's pitch according to an embodiment of the present invention, comprising:

[0052] Step S201, obtaining multiple pitches of each speaker, normalizing each pitch of each speaker to obtain an individual normalized pitch of each pitch, and inputting the multiple individual normalized pitches into an encoder of a text-to-speech model for encoding training;

[0053] Step S202, performing global normalization on each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and inputting the absolute normalized pitches into a decoder of a text-to-speech model for decoding audio feature training;

[0054] Step S203, using the trained encoder to predict the pitch of the text to be converted of the speaker to be predicted, and using the trained encoder and decoder to synthesize the audio features of the text to be converted based on the pitch.

[0055] Specifically, the present invention can be applied to electronic devices with processing capabilities, such as computers.

[0056] In the training phase, steps S201 and S202 are executed to train the text-to-speech (TTS) model. The TTS model includes an encoder and a decoder. The encoder is responsible for encoding the input text into high-dimensional features. The decoder is responsible for decoding the high-dimensional features in the encoder to generate audio features.

[0057] In the training phase, step S201 is first executed to obtain multiple pitches of each speaker, normalize each pitch of each speaker to obtain an individual normalized pitch of each pitch, and input the multiple individual normalized pitches into the encoder of the text-to-speech model for encoding training.

[0058] Among them, each speaker has multiple training texts for training and audio data corresponding to the training texts. The pitch corresponding to each speaker's different training texts can be extracted from the audio data. The individual normalized pitch is the pitch obtained by normalizing all the pitches of the speaker.

[0059] Specifically, taking two speakers speaker1 and speaker2 as examples, each pitch pitch1 of speaker1 is normalized to obtain the individual normalized pitch pitch corresponding to each pitch of speaker1 norm1 , normalize each pitch pitch2 of speaker2 to obtain the individual normalized pitch pitch corresponding to each pitch of speaker2 norm2 .

[0060] Then, a plurality of the individual normalized pitches are input into the encoder of the tts model for encoding training.

[0061] The encoding training of the encoder adopts the existing encoding training method, and the training text used for training and the corresponding individual normalized pitch are input into the encoder for encoding training, and the training text is encoded into high-dimensional features.

[0062] Then, step S202 is executed to globally normalize each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and the absolute normalized pitch is input into a decoder of a text-to-speech model for decoding audio feature training.

[0063] Specifically, each individual normalized pitch is globally normalized to obtain an absolute normalized pitch corresponding to each individual normalized pitch. The absolute normalized pitch is a pitch normalized by the absolute pitch obtained by globally normalizing the individual normalized pitch.

[0064] Then, the decoder decodes the absolute normalized pitch and high-dimensional features using an existing decoding method to synthesize audio features. The audio features are preferably Mel spectrograms.

[0065] The audio features generated by the encoder are compared with the audio data corresponding to the training text, and the model parameters of the encoder and / or decoder are corrected according to the comparison results to complete the model training.

[0066] In the prediction stage, step S203 is executed to use the trained encoder to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder synthesize the audio features of the text to be converted based on the pitch.

[0067] In some embodiments, the synthesizing of the audio features of the text to be converted based on the pitch by the trained encoder and decoder includes: encoding the text to be converted by the trained encoder to obtain high-dimensional features of the text to be converted, inputting the pitch and high-dimensional features of the text to be converted into the trained decoder, and decoding the high-dimensional features and the pitch by the trained decoder to synthesize the audio features.

[0068] Specifically, the trained encoder predicts the pitch of the text to be converted of the speaker to be predicted. For example, if the audio of speaker 1's timbre is to be synthesized, the encoder predicts the pitch of speaker 1. At the same time, the encoder encodes the text to be converted to obtain the corresponding high-dimensional features.

[0069] The decoder then uses the existing decoding and synthesis method to decode the high-dimensional features and pitch of the text and synthesize the audio features.

[0070] In the training stage, the present invention normalizes the pitch of each speaker to obtain an individual normalized pitch, and inputs a plurality of the individual normalized pitches into the encoder of the text-to-speech model for encoding training; globally normalizes each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and inputs the absolute normalized pitch into the decoder of the text-to-speech model for decoding audio feature training; in the prediction stage, the trained encoder is used to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder synthesize the audio features of the text to be converted based on the pitch. The present invention performs normalization according to different speakers, ensures that the pitch distribution of different speakers is the same, thereby improving the accuracy of the encoder in predicting pitch. At the same time, the decoder performs back-normalization and global normalization to obtain the absolute normalized pitch, ensuring that the decoder can synthesize the audio features with accurate pitch when decoding.

[0071] like Figure 3 FIG. 1 is a flowchart of a method for predicting a speaker's pitch according to another embodiment of the present invention, comprising:

[0072] Step S301, obtaining multiple pitches of each speaker.

[0073] Step S302: Calculate the pitch mean and pitch variance of each speaker.

[0074] Step S303: Calculate the individual normalized pitch of each pitch of each speaker based on the pitch mean and pitch variance of each speaker.

[0075] In one embodiment, the step of calculating the individual normalized pitch of each pitch of each speaker based on the pitch mean and pitch variance of each speaker includes:

[0076] Calculate the individual normalized pitch corresponding to the jth pitch of the i-th speaker as:

[0077] pitch normi,j =(pitch i,j -mean i ) / std i ;

[0078] Among them, pitch normi,j is the individual normalized pitch corresponding to the jth pitch of the ith speaker, pitch i,j is the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker.

[0079] Step S304: input the plurality of individual normalized pitches into an encoder of a text-to-speech model for encoding training.

[0080] Step S305 , calculating the total mean of the pitches of all speakers as the global pitch mean, and calculating the total variance of the pitches of all speakers as the global pitch variance.

[0081] Step S306, denormalizing each of the individual normalized pitches, and then performing global normalization based on the global pitch mean and the global pitch variance to obtain an absolute normalized pitch.

[0082] In one embodiment, the denormalizing each of the individual normalized pitches and then performing global normalization based on a global pitch mean and a global pitch variance to obtain an absolute normalized pitch comprises:

[0083] Calculate the absolute normalized pitch corresponding to the jth pitch of the i-th speaker as:

[0084] pitch fi,i =((pitch normi,j *std i +mean i )-mean) / std;

[0085] Among them, pitch fi is the absolute normalized pitch corresponding to the jth pitch of the ith speaker, pitch normi is the individual normalized pitch corresponding to the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker, mean is the global pitch mean, and std is the global pitch variance.

[0086] Step S307: input the absolute normalized pitch into a decoder of a text-to-speech model for decoding audio feature training.

[0087] Step S308 : using the trained encoder to predict the individual normalized pitch of the speaker to be predicted based on the text to be converted of the speaker to be predicted.

[0088] Step S309, obtaining the pitch mean and pitch variance of the speaker to be predicted, calculating the absolute normalized pitch of the individual normalized pitch of the speaker to be predicted based on the pitch mean and pitch variance of the speaker to be predicted, and using the absolute normalized pitch of the speaker to be predicted as the pitch of the text to be converted of the speaker to be predicted.

[0089] In one of the embodiments, the step of calculating the absolute normalized pitch of the individual normalized pitch of the speaker to be predicted based on the pitch mean and pitch variance of the speaker to be predicted includes:

[0090] Finding the pitch mean of the speaker to be predicted, the pitch variance of the speaker to be predicted, the global pitch mean and the global pitch variance;

[0091] The absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated as:

[0092] pitch p =((pitch normp *std p +mean p )-mean) / std;

[0093] Among them, pitch p is the absolute normalized pitch of the speaker to be predicted, pitch normp is the individual normalized pitch of the speaker to be predicted predicted by the trained decoder, mean p is the mean pitch of the speaker to be predicted, std p is the pitch variance of the speaker to be predicted, mean is the global pitch mean, and std is the global pitch variance.

[0094] Step S310: The trained encoder and decoder synthesize the audio features of the text to be converted based on the pitch.

[0095] Specifically, step S301 to step S307 is the tts model training stage.

[0096] In the training phase, step S301 is first performed to obtain multiple pitches of each speaker.

[0097] Each speaker has multiple training texts for training and audio data corresponding to the training texts. The pitch corresponding to the different training texts of each speaker can be extracted from the audio data.

[0098] Then, step S302 is performed to calculate the pitch mean and pitch variance of each speaker.

[0099] Since each speaker has multiple pitches for different training texts, the mean of all pitches of each speaker is calculated as the pitch mean, and the variance of all pitches of each speaker is calculated as the pitch variance.

[0100] As an example, calculate the mean of all pitches for speaker1 1 , calculate the variance std of all pitches of speech1 1 . Calculate the mean of all pitches for speaker2 2 , calculate the variance std of all pitches of speech2 2 For example, speaker 1 has 10 pitches and speaker 2 has 15 pitches, then mean 1 and std 1 Using the 10 pitches of speaker 1, mean 2 and std 2 The 15 pitches of speaker2 are used for calculation.

[0101] Afterwards, step S303 is performed to calculate the individual normalized pitch of each pitch of each speaker based on the pitch mean and pitch variance of each speaker.

[0102] After calculating the pitch mean and pitch variance of each speaker, since a speaker has multiple pitches, the corresponding individual normalized pitch is calculated for each pitch of each speaker. The calculated individualized normalized pitch is associated with the pitch and the training text corresponding to the pitch.

[0103] In one embodiment, the step of calculating the individual normalized pitch of each pitch of each speaker based on the pitch mean and pitch variance of each speaker includes:

[0104] Calculate the individual normalized pitch corresponding to the jth pitch of the i-th speaker as:

[0105] pitch normi,j =(pitch i,j -mean i ) / std i ;

[0106] Among them, pitch normi,j is the individual normalized pitch corresponding to the jth pitch of the ith speaker, pitch i,j is the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker.

[0107] Since each speaker has one or more pitches, for the jth pitch of the ith speaker, the corresponding individual normalized pitch is calculated as:

[0108] pitch norm,i,j =(pitch i,j -mean i ) / std i ;

[0109] Among them, pitch norm,i,j is the individual normalized pitch corresponding to the jth pitch of the ith speaker, pitch i,j is the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker.

[0110] As an example, normalize the pitch of two speakers separately:

[0111] pitch norm1,a =(pitch 1,a -mean 1 ) / std 1

[0112] pitch norm2,b =(pitch 2,b -mean 1 0 / std 2

[0113] Among them, pitch 1,a is the pitch of speaker 1, pitch norm1,a For pitch 1,a The corresponding individual normalized pitch, pitch 2,b is the bth pitch of speaker 2, pitch norm2,b For pitch 2,b The corresponding individual normalized pitch.

[0114] Then, step S304 is executed to input the plurality of individual normalized pitches into an encoder of a text-to-speech model for encoding training.

[0115] Specifically, the encoding training of the encoder adopts the existing encoding training method, and the training text used for training and the corresponding individual normalized pitch are input into the encoder for encoding training, and the training text is encoded into high-dimensional features.

[0116] Then, step S305 is executed to calculate the total mean of the pitches of all speakers as the global pitch mean, and calculate the total variance of the pitches of all speakers as the global pitch variance.

[0117] Specifically, the total mean of all pitches of all speakers is calculated as the global pitch mean, and the total variance of all pitches of all speakers is calculated as the global pitch variance. As an example, speaker 1 has 10 pitches and speaker 2 has 15 pitches, then mean 1 and std 1 Using the 10 pitches of speaker 1, mean 2 and std 2 The 15 pitches of speaker2 are used for calculation, while the mean and std are calculated using a total of 25 pitches of speaker1 and speaker2.

[0118] It should be noted that step S305 does not have to be executed after step S304, and step S305 can be executed at any position after step S301 and before step S306.

[0119] Then, step S306 is executed to perform denormalization on each of the individual normalized pitches, and then perform global normalization based on the global pitch mean and the global pitch variance to obtain an absolute normalized pitch.

[0120] In one embodiment, the denormalizing each of the individual normalized pitches and then performing global normalization based on a global pitch mean and a global pitch variance to obtain an absolute normalized pitch comprises:

[0121] Calculate the absolute normalized pitch corresponding to the jth pitch of the i-th speaker as:

[0122] pitch fi,j =((pitch normi,j *std 1 +mean i )-mean) / std;

[0123] Among them, pitch fi is the absolute normalized pitch corresponding to the jth pitch of the ith speaker, pitch normi is the individual normalized pitch corresponding to the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker, mean is the global pitch mean, and std is the global pitch variance.

[0124] Specifically, since each individual normalized pitch corresponds to a certain pitch of a speaker, denormalization is performed based on the pitch mean and pitch variance of the speaker corresponding to the individual normalized pitch, and then global normalization is performed based on the global pitch mean and global pitch variance to obtain the absolute normalized pitch.

[0125] As an example, the corresponding Jedi normalized pitches are calculated for the individual normalized pitches of two speakers:

[0126] pitch f1,a =((pitch norm1,a *std 1 +mean 1 )-mean) / std

[0127] pitch f2,b =((pitch norm2,b *std 2 +mean 2 0-mean) / std

[0128] Among them, pitch fi,a is the absolute normalized pitch corresponding to the ath pitch of the ith speaker, pitch normi is the individual normalized pitch corresponding to the ath pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker, mean is the global pitch mean, and std is the global pitch variance.

[0129] This embodiment provides a specific calculation method for absolute normalized pitch.

[0130] Then, step S307 is performed to input the absolute normalized pitch into a decoder of a text-to-speech model for decoding audio feature training.

[0131] Specifically, the decoder decodes the high-dimensional features and absolute normalized pitch output by the encoder using an existing decoding method to generate audio features. The audio features are preferably Mel-spectrograms.

[0132] The audio features generated by the encoder are compared with the audio data corresponding to the training text, and the model parameters of the encoder and / or decoder are corrected according to the comparison results to complete the model training.

[0133] Step S308 to step S310 is the TTS model prediction stage.

[0134] In the prediction stage, step S308 is first performed to use the trained encoder to predict the individual normalized pitch of the speaker to be predicted based on the text to be converted of the speaker to be predicted.

[0135] Specifically, in the prediction stage, if it is necessary to synthesize audio of speaker 1's timbre, but there is no required pitch of the synthesized audio, then the decoder's additional pitch prediction module uses the existing prediction method to predict the individual normalized pitch of speaker 1 based on the text to be converted.

[0136] Then, step S309 is performed to obtain the pitch mean and pitch variance of the speaker to be predicted, calculate the absolute normalized pitch of the individual normalized pitch of the speaker to be predicted based on the pitch mean and pitch variance of the speaker to be predicted, and use the absolute normalized pitch of the speaker to be predicted as the pitch of the text to be converted of the speaker to be predicted.

[0137] Specifically, the pitch mean and pitch variance of the speaker to be predicted can be found from the training data, and based on the pitch mean and pitch variance of the speaker to be predicted, the absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated. After the individual normalized pitch of the speaker to be predicted is converted into the absolute normalized pitch, the absolute normalized pitch of the speaker to be predicted is input into the decoder as the pitch of the text to be converted of the speaker to be predicted.

[0138] In one of the embodiments, the step of calculating the absolute normalized pitch of the individual normalized pitch of the speaker to be predicted based on the pitch mean and pitch variance of the speaker to be predicted includes:

[0139] Finding the pitch mean of the speaker to be predicted, the pitch variance of the speaker to be predicted, the global pitch mean and the global pitch variance;

[0140] The absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated as:

[0141] pitch p =((pitch normp *std p +mean p )-mean) / std;

[0142] Among them, pitch p is the absolute normalized pitch of the speaker to be predicted, pitch normp is the individual normalized pitch of the speaker to be predicted predicted by the trained decoder, mean p is the mean pitch of the speaker to be predicted, std pis the pitch variance of the speaker to be predicted, mean is the global pitch mean, and std is the global pitch variance.

[0143] Specifically, based on the identifier of the speaker to be predicted, such as speaker ID, the pitch mean, pitch variance, global pitch mean and global pitch variance of the speaker with the same identifier as the speaker to be predicted are searched from the training data.

[0144] For example, if the speaker to be predicted is speaker1, then find the mean pitch of speaker1 from the training data. 1 , pitch variance std 1 , global pitch mean and global pitch variance std.

[0145] Then, based on the pitch mean and pitch variance of the speaker to be predicted, the individual normalized pitch of the speaker to be predicted predicted by the encoder is denormalized, and then globally normalized based on the global pitch mean and global pitch variance to obtain the absolute normalized pitch of the speaker to be predicted.

[0146] Finally, step S310 is executed, where the trained encoder and decoder synthesize the audio features of the text to be converted based on the pitch.

[0147] The encoder module of the tts of this embodiment performs normalization according to different speakers to ensure that the pitch distribution of different speakers is the same, thereby improving the accuracy of the encoder in predicting the pitch. At the same time, in the decoder module, the pitch predicted by the encoder is denormalized, and then globally normalized according to the total mean and variance to obtain the pitch after absolute pitch normalization, ensuring that the decoder can synthesize audio features with accurate pitch when decoding.

[0148] like Figure 4 The schematic diagram of the best embodiment of the present invention is shown in FIG.

[0149] During the training phase, the mean and variance of the pitch are calculated for speaker 1 and speaker 2 respectively. 1 , std 1 ,mean 2 ,std 2 , and calculate the mean and variance of the overall pitch mean, std. In the encoder module, one of the pitches of speaker1 is 1 , and one of the pitches of speaker 2 2 For example, for pitch 1 and pitch 2 Normalize them separately and input them into the encoder 41:

[0150] pitch norm1 =(pitch 1 -mean 1 ) / std 1

[0151] pitch norm2 =(pitch 2 -mean 1 ) / std 2

[0152] Among them, pitch norm1 For pitch 1 The corresponding individual normalized pitch, pitch norm2 For pitch 2 The corresponding individual normalized pitch.

[0153] In the decoder module, the individual normalized pitches of the two speakers are denormalized and then globally normalized and input into the decoder 42. This ensures that the pitch input to the decoder is absolute pitch, thereby decoding the audio features with accurate pitch:

[0154] pitch f1 =((pitch norm1 *std 1 +mean 1 )-mean) / std

[0155] pitch f2 =((pitch norm2 *std 2 +mean 2 )-mean) / std

[0156] Among them, pitch f1 For pitch norm1 The corresponding absolute normalized pitch, pitch f2 For pitch norm2 The corresponding absolute normalized pitch.

[0157] In the prediction stage, if you need to synthesize the audio of speaker 1, but there is no pitch to synthesize the audio, the encoder needs to predict the individual normalized pitch of speaker 1. norm1 , the encoder uses an additional pitch prediction module to predict. For example, the encoder's Variance Adaptor predicts the normalized pitch of speaker 1 based on the input speaker id and text. norm1, and denormalize according to the pitch mean, pitch variance of speaker 1 and the overall global pitch mean and global pitch variance, and then normalize according to the overall mean and variance to obtain the absolute normalized pitch of speaker 1 after absolute pitch normalization.

[0158] The trained encoder then encodes the text to be converted to obtain high-dimensional features of the text to be converted, and inputs the pitch and high-dimensional features of the text to be converted into the trained decoder. The trained decoder uses the existing decoding and synthesis method to decode the high-dimensional features and the pitch to synthesize audio features.

[0159] like Figure 5 The figure shows a schematic diagram of individual normalized pitch predicted by an encoder in an example of the present invention. In the figure, the horizontal axis is the audio frame and the vertical axis is the individual normalized pitch. It can be seen that the values ​​are basically on both sides of the mean value 0, are evenly distributed, and have no abnormal points.

[0160] Based on the same inventive concept, Figure 6 FIG. 1 is a schematic diagram of a speaker pitch prediction device according to an embodiment of the present invention, comprising:

[0161] The encoder training module 601 is used to obtain multiple pitches of each speaker, normalize each pitch of each speaker to obtain an individual normalized pitch of each pitch, and input the multiple individual normalized pitches into the encoder of the text-to-speech model for encoding training;

[0162] A decoder training module 602 is used to perform global normalization on each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and input the absolute normalized pitches into a decoder of a text-to-speech model for decoding audio feature training;

[0163] The prediction module 603 is used to use the trained encoder to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder synthesize the audio features of the text to be converted based on the pitch.

[0164] In the training stage, the present invention normalizes the pitch of each speaker to obtain an individual normalized pitch, and inputs a plurality of the individual normalized pitches into the encoder of the text-to-speech model for encoding training; globally normalizes each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and inputs the absolute normalized pitch into the decoder of the text-to-speech model for decoding audio feature training; in the prediction stage, the trained encoder is used to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder synthesize the audio features of the text to be converted based on the pitch. The present invention performs normalization according to different speakers, ensures that the pitch distribution of different speakers is the same, thereby improving the accuracy of the encoder in predicting pitch. At the same time, the decoder performs back-normalization and global normalization to obtain the absolute normalized pitch, ensuring that the decoder can synthesize the audio features with accurate pitch when decoding.

[0165] In one embodiment, normalizing the pitch of each speaker to obtain an individual normalized pitch includes:

[0166] Calculate the mean pitch and pitch variance of each speaker respectively;

[0167] Based on the pitch mean and pitch variance of each speaker, the individual normalized pitch of each pitch of each speaker is calculated.

[0168] In one embodiment, the step of calculating the individual normalized pitch of each pitch of each speaker based on the pitch mean and pitch variance of each speaker includes:

[0169] Calculate the individual normalized pitch corresponding to the jth pitch of the i-th speaker as:

[0170] pitch normi,j =(pitch i,j -mean i ) / std i ;

[0171] Among them, pitch normi,j is the individual normalized pitch corresponding to the jth pitch of the ith speaker, pitch i,j is the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker.

[0172] In one embodiment, the globally normalizing each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches includes:

[0173] Calculate the total mean of the pitch of all speakers as the global pitch mean, and calculate the total variance of the pitch of all speakers as the global pitch variance;

[0174] Each of the individual normalized pitches is denormalized, and then globally normalized based on the global pitch mean and the global pitch variance to obtain an absolute normalized pitch.

[0175] In one embodiment, the denormalizing each of the individual normalized pitches and then performing global normalization based on a global pitch mean and a global pitch variance to obtain an absolute normalized pitch comprises:

[0176] In one embodiment, the denormalizing each of the individual normalized pitches and then performing global normalization based on a global pitch mean and a global pitch variance to obtain an absolute normalized pitch comprises:

[0177] Calculate the absolute normalized pitch corresponding to the jth pitch of the i-th speaker as:

[0178] pitch fi,j =((pitch normi,j *std 1 +mean i )-mean) / std;

[0179] Among them, pitch fi is the absolute normalized pitch corresponding to the jth pitch of the ith speaker, pitch normi is the individual normalized pitch corresponding to the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker, mean is the global pitch mean, and std is the global pitch variance.

[0180] In one embodiment, the step of using the trained encoder to predict the pitch of the text to be converted of the speaker to be predicted includes:

[0181] Using the trained encoder, based on the text to be converted of the speaker to be predicted, predict the individual normalized pitch of the speaker to be predicted;

[0182] The pitch mean and pitch variance of the speaker to be predicted are obtained, and based on the pitch mean and pitch variance of the speaker to be predicted, an absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated, and the absolute normalized pitch of the speaker to be predicted is used as the pitch of the text to be converted of the speaker to be predicted.

[0183] In one of the embodiments, the step of calculating the absolute normalized pitch of the individual normalized pitch of the speaker to be predicted based on the pitch mean and pitch variance of the speaker to be predicted includes:

[0184] Finding the pitch mean of the speaker to be predicted, the pitch variance of the speaker to be predicted, the global pitch mean and the global pitch variance;

[0185] The absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated as:

[0186] pitch p =((pitch normp *std p +mean p )-mean) / std;

[0187] Among them, pitch p is the absolute normalized pitch of the speaker to be predicted, pitch normp is the individual normalized pitch of the speaker to be predicted predicted by the trained decoder, mean p is the mean pitch of the speaker to be predicted, std p is the pitch variance of the speaker to be predicted, mean is the global pitch mean, and std is the global pitch variance.

[0188] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0189] like Figure 7 The figure shows a hardware structure diagram of an electronic device of the present invention, including:

[0190] at least one processor 701; and,

[0191] A memory 702 that is communicatively connected to at least one of the processors 701; wherein,

[0192] The memory 702 stores instructions that can be executed by at least one of the processors. The instructions are executed by at least one of the processors to enable the at least one of the processors to perform the speaker pitch prediction method as described above.

[0193] Figure 7 A processor 701 is taken as an example.

[0194] The electronic device may further include: an input device 703 and a display device 704 .

[0195] The processor 701, the memory 702, the input device 703 and the display device 704 may be connected via a bus or other means, and the figure takes the connection via a bus as an example.

[0196] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the speaker pitch prediction method in the embodiment of the present application, for example, Figure 1 , Figure 2 , Figure 3 The processor 701 executes various functional applications and data processing by running the non-volatile software programs, instructions and modules stored in the memory 702, that is, the speaker pitch prediction method in the above embodiment is implemented.

[0197] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the speaker pitch prediction method, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the device for executing the speaker pitch prediction method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0198] The input device 703 can receive user clicks and generate signal inputs related to user settings and function controls of the speaker pitch prediction method. The display device 704 can include display devices such as a display screen.

[0199] The one or more modules are stored in the memory 702 , and when executed by the one or more processors 701 , the speaker pitch prediction method in any of the above method embodiments is executed.

[0200] In the training stage, the present invention normalizes the pitch of each speaker to obtain an individual normalized pitch, and inputs a plurality of the individual normalized pitches into the encoder of the text-to-speech model for encoding training; globally normalizes each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and inputs the absolute normalized pitch into the decoder of the text-to-speech model for decoding audio feature training; in the prediction stage, the trained encoder is used to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder synthesize the audio features of the text to be converted based on the pitch. The present invention performs normalization according to different speakers, ensures that the pitch distribution of different speakers is the same, thereby improving the accuracy of the encoder in predicting pitch. At the same time, the decoder performs back-normalization and global normalization to obtain the absolute normalized pitch, ensuring that the decoder can synthesize the audio features with accurate pitch when decoding.

[0201] An embodiment of the present invention provides a storage medium, wherein the storage medium stores computer instructions. When a computer executes the computer instructions, the storage medium is used to execute all steps of the speaker pitch prediction method as described above.

[0202] In the context of the present disclosure, a storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. The storage medium may be a machine-readable signal medium or a machine-readable storage medium. Alternatively, the storage medium may be a non-temporary computer-readable storage medium, for example, a non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0203] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for predicting a speaker's pitch, characterized in that: include: Acquire multiple pitches of each speaker, normalize each pitch of each speaker to obtain an individual normalized pitch of each pitch, and input the multiple individual normalized pitches into an encoder of a text-to-speech model for encoding training; Performing global normalization on each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and inputting the absolute normalized pitches into a decoder of a text-to-speech model for decoding audio feature training; The trained encoder is used to predict the pitch of the text to be converted of the speaker to be predicted, and the trained encoder and decoder are used to synthesize the audio features of the text to be converted based on the pitch; The globally normalizing each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches includes: Calculate the total mean of the pitch of all speakers as the global pitch mean, and calculate the total variance of the pitch of all speakers as the global pitch variance; Denormalizing each of the individual normalized pitches, and then performing global normalization based on a global pitch mean and a global pitch variance to obtain an absolute normalized pitch; Denormalizing each of the individual normalized pitches, and then performing global normalization based on the global pitch mean and the global pitch variance to obtain an absolute normalized pitch, includes: Calculate the absolute normalized pitch corresponding to the jth pitch of the i-th speaker as: pitch fi,j =((pitch normi,j *std i +mean i )-mean) / std; Among them, pitch fi is the absolute normalized pitch corresponding to the jth pitch of the ith speaker, pitch normi is the individual normalized pitch corresponding to the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker, mean is the global pitch mean, and std is the global pitch variance.

2. The speaker pitch prediction method according to claim 1, characterized in that: The step of normalizing the pitch of each speaker to obtain an individual normalized pitch includes: Calculate the mean pitch and pitch variance of each speaker respectively; Based on the pitch mean and pitch variance of each speaker, the individual normalized pitch of each pitch of each speaker is calculated.

3. The speaker pitch prediction method according to claim 2, characterized in that: The step of calculating the individual normalized pitch of each pitch of each speaker based on the pitch mean and pitch variance of each speaker comprises: Calculate the individual normalized pitch corresponding to the jth pitch of the i-th speaker as: pitch normi,j =(pitch i,j -mean i ) / std i ; Among them, pitch normi,j is the individual normalized pitch corresponding to the jth pitch of the ith speaker, pitch i,j is the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker.

4. The speaker pitch prediction method according to claim 1, characterized in that: The method of using the trained encoder to predict the pitch of the text to be converted of the speaker to be predicted includes: Using the trained encoder, based on the text to be converted of the speaker to be predicted, predict the individual normalized pitch of the speaker to be predicted; The pitch mean and pitch variance of the speaker to be predicted are obtained, and based on the pitch mean and pitch variance of the speaker to be predicted, an absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated, and the absolute normalized pitch of the speaker to be predicted is used as the pitch of the text to be converted of the speaker to be predicted.

5. The speaker pitch prediction method according to claim 4, characterized in that: The step of calculating the absolute normalized pitch of the individual normalized pitch of the speaker to be predicted based on the pitch mean and pitch variance of the speaker to be predicted includes: Finding the pitch mean of the speaker to be predicted, the pitch variance of the speaker to be predicted, the global pitch mean and the global pitch variance; The absolute normalized pitch of the individual normalized pitch of the speaker to be predicted is calculated as: pitch p =((pitch normp *std p +mean p )-mean) / std; Among them, pitch p is the absolute normalized pitch of the speaker to be predicted, pitch normp is the individual normalized pitch of the speaker to be predicted predicted by the trained decoder, mean p is the mean pitch of the speaker to be predicted, std p is the pitch variance of the speaker to be predicted, mean is the global pitch mean, and std is the global pitch variance.

6. A speaker pitch prediction device, characterized in that: include: An encoder training module, used to obtain multiple pitches of each speaker, normalize each pitch of each speaker to obtain an individual normalized pitch of each pitch, and input the multiple individual normalized pitches into an encoder of a text-to-speech model for encoding training; A decoder training module, used to perform global normalization on each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches, and input the absolute normalized pitches into a decoder of a text-to-speech model for decoding audio feature training; A prediction module, configured to use a trained encoder to predict the pitch of a text to be converted of a speaker to be predicted, and synthesize audio features of the text to be converted based on the pitch by the trained encoder and decoder; The globally normalizing each of the individual normalized pitches to obtain an absolute normalized pitch corresponding to each of the individual normalized pitches includes: Calculate the total mean of the pitch of all speakers as the global pitch mean, and calculate the total variance of the pitch of all speakers as the global pitch variance; Denormalizing each of the individual normalized pitches, and then performing global normalization based on a global pitch mean and a global pitch variance to obtain an absolute normalized pitch; Denormalizing each of the individual normalized pitches, and then performing global normalization based on the global pitch mean and the global pitch variance to obtain an absolute normalized pitch, includes: Calculate the absolute normalized pitch corresponding to the jth pitch of the i-th speaker as: pitch fi,j =((pitch normi,j *std i +mean i )-mean) / std; Among them, pitch fi is the absolute normalized pitch corresponding to the jth pitch of the ith speaker, pitch normi is the individual normalized pitch corresponding to the jth pitch of the ith speaker, mean i is the mean pitch of the i-th speaker, std i is the pitch variance of the ith speaker, mean is the global pitch mean, and std is the global pitch variance.

7. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors to enable at least one of the processors to execute the speaker pitch prediction method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores computer instructions, and when a computer executes the computer instructions, it is used to execute all steps of the speaker pitch prediction method according to any one of claims 1 to 5.

Images