Generation method and device of acoustic model, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an acoustic model generation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. Comprising text content, audio pediatric data corresponding to the text content, text rhythm features corresponding to the text content and acoustic features corresponding to the audio pediatric data, and the acoustic model comprises a secondary rhythm representation correction module and an acoustic processing unit; inputting the text content, the audio data, the acoustic features and the text rhythm features into a second-level rhythm representation correction module for correction, and performing parameter tuning on the second-level rhythm representation correction module based on a correction result to obtain a second-level rhythm representation correction module after tuning; performing model training according to the text content, the audio data, the acoustic features and the optimized secondary rhythm representation correction module to obtain a loss function; and performing parameter optimization on the acoustic processing unit according to the loss function to obtain an acoustic model meeting a preset condition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for generating an acoustic model, a device for generating an acoustic model, an electronic device, and a computer-readable storage medium. Background Art

[0002] Speech synthesis is a technology that converts text into speech. It usually requires the following processes: front-end text processing, acoustic model prediction of acoustic features (such as Mel spectrum), and vocoder conversion of acoustic features into speech. Among them, the acoustic model is a key step in modeling the content, timbre, and acoustic rhythm of the synthesized audio. It usually requires input of phoneme-level text features to specify content information, speaker information to specify the synthesized timbre, and rhythmic structure label information to ensure the controllability of rhythmic pauses. However, in the process of building the corresponding acoustic model, there are problems such as the inability to reflect the different rhythmic styles of different speakers and the inability to ensure accurate correspondence with the real audio. Summary of the invention

[0003] An embodiment of the present invention provides a method, device, electronic device and computer-readable storage medium for generating an acoustic model to solve or partially solve the problem that the model constructed in the process of constructing the acoustic model cannot reflect the different rhythmic styles of different speakers and cannot ensure accurate correspondence with the real audio.

[0004] The embodiment of the present invention discloses a method for generating an acoustic model, comprising:

[0005] Acquire training data for acoustic model training, the training data at least including text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, the acoustic model at least including a secondary prosodic representation correction module and an acoustic processing unit;

[0006] Inputting the text content, the audio pair data, the acoustic features and the text prosody features into the secondary prosody representation correction module to correct the prosody label, and optimizing the parameters of the secondary prosody representation correction module based on the correction result to obtain an optimized secondary prosody representation correction module;

[0007] Performing model training according to the text content, the audio pair data, the acoustic features, and the tuned secondary prosody representation correction module to obtain at least two loss functions;

[0008] The acoustic processing unit is parameter-tuned according to at least two of the loss functions to obtain an acoustic model that meets preset conditions, and the acoustic model is used to synthesize the target audio corresponding to the corresponding speaker.

[0009] In some feasible implementations, the text prosody features include at least a first-class prosody label, the first-class prosody label includes at least a predicted first-class prosody label, a predicted second-class prosody label, a predicted third-class prosody label, and a real second-class prosody label, the text content, the audio pair data, the acoustic features, and the text prosody features are input into the second-class prosody representation correction module to correct the prosody label, and the second-class prosody representation correction module is parameter-tuned based on the correction result to obtain the tuned second-class prosody representation correction module, including:

[0010] Inputting the text content, the audio pair data, the acoustic features, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the secondary prosody representation correction module for prediction to obtain a corresponding prediction result;

[0011] Comparing the prediction result with the true secondary prosody label to determine the prosody label prediction loss for the secondary prosody representation correction module;

[0012] The prosody label prediction loss is used to perform parameter tuning on the secondary prosody representation correction module until the prosody label prediction loss meets a preset condition, thereby obtaining a tuned secondary prosody representation correction module.

[0013] In some feasible implementations, the acoustic feature at least includes a speaker identifier, and the text content, the audio pair data, the acoustic feature, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label are input into the secondary prosody representation correction module for prediction to obtain a corresponding prediction result, including:

[0014] Obtaining phoneme features corresponding to the audio pair data;

[0015] The text content, the speaker identifier, the phoneme feature, the predicted primary prosody label, the predicted secondary prosody label and the predicted tertiary prosody label are input into the secondary prosody representation correction module for prediction to obtain corresponding prediction results.

[0016] In some feasible implementations, the acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor, and the model training is performed according to the text content, the audio pair data, the acoustic features, and the tuned secondary prosody representation correction module to obtain at least two loss functions, including:

[0017] Inputting the text content, the audio pair data, the acoustic features, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the optimized secondary prosody representation correction module to obtain a corrected secondary prosody label;

[0018] The text content, the audio pair data, the acoustic features, the predicted primary prosody label, the corrected secondary prosody label, and the predicted tertiary prosody label are used to perform representation combination to obtain a corresponding first content representation;

[0019] Inputting the first content representation into the encoder for encoding to obtain a corresponding first intermediate representation;

[0020] Inputting the first intermediate representation into the variable feature predictor for prediction, obtaining a first variable feature related to speech, and comparing the variable feature with a preset first real content, and calculating a corresponding first variable feature prediction loss;

[0021] The variable feature is input into the decoder for decoding processing to obtain a corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with a preset second real content to calculate the corresponding first Mel spectrum prediction loss.

[0022] In some feasible implementations, the text prosody features include at least a second type of prosody labels, the second type of prosody labels include at least a predicted primary prosody label, a predicted secondary prosody label, and a predicted tertiary prosody label, and the model training is performed according to the text content, the audio pair data, the acoustic features, and the tuned secondary prosody representation correction module to obtain at least two loss functions, including:

[0023] The text content, the audio pair data, the acoustic feature, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label are used to perform representation combination to obtain a corresponding second content representation;

[0024] Inputting the second content representation into the encoder for encoding to obtain a corresponding second intermediate representation;

[0025] Inputting the second intermediate representation into the variable feature predictor for prediction, obtaining a second variable feature related to speech, and comparing the variable feature with a preset first real content, and calculating a corresponding second variable feature prediction loss;

[0026] The variable feature is input into the decoder for decoding processing to obtain a corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with a preset second real content to calculate a corresponding second Mel spectrum prediction loss.

[0027] In some feasible implementations, the step of tuning parameters of the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions includes:

[0028] Optimizing parameters of the acoustic processing unit according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until an iteration condition is met, thereby obtaining an optimized acoustic processing unit;

[0029] Based on the tuned secondary prosody representation correction module and the tuned acoustic processing unit, a corresponding acoustic model is constructed.

[0030] In some feasible implementations, the acoustic model further includes at least an acoustic processing unit, and the method further includes:

[0031] Acquire a target phoneme sequence, a target prosody label, and a target speaker identifier, wherein the target prosody label at least includes a predicted primary prosody label, a predicted secondary prosody label, and a predicted tertiary prosody label;

[0032] Inputting the target phoneme sequence, the target speaker identifier, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the secondary prosody representation correction module to correct the prosody label, and obtain a corrected secondary prosody label;

[0033] Combining the target phoneme sequence, the target speaker identifier, the predicted primary prosody label, the corrected secondary prosody label, and the predicted tertiary prosody label to obtain a target content representation;

[0034] The target content representation is input into the acoustic processing unit for audio synthesis to obtain target audio corresponding to the target factor sequence.

[0035] The embodiment of the present invention further discloses a device for generating an acoustic model, comprising:

[0036] A data acquisition module, used to acquire training data for acoustic model training, wherein the training data at least includes text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, and the acoustic model at least includes a secondary prosodic representation correction module and an acoustic processing unit;

[0037] A correction module, used for inputting the text content, the audio pair data, the acoustic features and the text prosody features into the secondary prosody representation correction module to correct the prosody label, and performing parameter tuning on the secondary prosody representation correction module based on the correction result to obtain a tuned secondary prosody representation correction module;

[0038] A training module, used for performing model training according to the text content, the audio pair data, the acoustic features and the tuned secondary prosody representation correction module to obtain at least two loss functions;

[0039] A tuning module is used to perform parameter tuning on the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions, and the acoustic model is used to synthesize the target audio corresponding to the corresponding speaker.

[0040] In some feasible implementations, the text prosodic features include at least a first type of prosodic label, the first type of prosodic label includes at least a predicted first level prosodic label, a predicted second level prosodic label, a predicted third level prosodic label, and a real second level prosodic label, and the correction module is specifically used for:

[0041] Inputting the text content, the audio pair data, the acoustic features, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the secondary prosody representation correction module for prediction to obtain a corresponding prediction result;

[0042] Comparing the prediction result with the true secondary prosody label to determine the prosody label prediction loss for the secondary prosody representation correction module;

[0043] The prosody label prediction loss is used to perform parameter tuning on the secondary prosody representation correction module until the prosody label prediction loss meets a preset condition, thereby obtaining a tuned secondary prosody representation correction module.

[0044] In some feasible implementations, the acoustic feature includes at least a speaker identifier, and the correction module is specifically configured to:

[0045] Obtaining phoneme features corresponding to the audio pair data;

[0046] The text content, the speaker identifier, the phoneme feature, the predicted primary prosody label, the predicted secondary prosody label and the predicted tertiary prosody label are input into the secondary prosody representation correction module for prediction to obtain corresponding prediction results.

[0047] In some feasible implementations, the acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor, and the training module is specifically used for:

[0048] Inputting the text content, the audio pair data, the acoustic features, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the optimized secondary prosody representation correction module to obtain a corrected secondary prosody label;

[0049] The text content, the audio pair data, the acoustic features, the predicted primary prosody label, the corrected secondary prosody label, and the predicted tertiary prosody label are used to perform representation combination to obtain a corresponding first content representation;

[0050] Inputting the first content representation into the encoder for encoding to obtain a corresponding first intermediate representation;

[0051] Inputting the first intermediate representation into the variable feature predictor for prediction, obtaining a first variable feature related to speech, and comparing the variable feature with a preset first real content, and calculating a corresponding first variable feature prediction loss;

[0052] The variable feature is input into the decoder for decoding processing to obtain a corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with a preset second real content to calculate the corresponding first Mel spectrum prediction loss.

[0053] In some feasible implementations, the text prosodic features include at least a second type of prosodic label, the second type of prosodic label includes at least a predicted primary prosodic label, a predicted secondary prosodic label, and a predicted tertiary prosodic label, and the training module is specifically used for:

[0054] The text content, the audio pair data, the acoustic feature, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label are used to perform representation combination to obtain a corresponding second content representation;

[0055] Inputting the second content representation into the encoder for encoding to obtain a corresponding second intermediate representation;

[0056] Inputting the second intermediate representation into the variable feature predictor for prediction, obtaining a second variable feature related to speech, and comparing the variable feature with a preset first real content, and calculating a corresponding second variable feature prediction loss;

[0057] The variable feature is input into the decoder for decoding processing to obtain a corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with a preset second real content to calculate a corresponding second Mel spectrum prediction loss.

[0058] In some feasible implementations, the tuning module is specifically used to:

[0059] Optimizing parameters of the acoustic processing unit according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until an iteration condition is met, thereby obtaining an optimized acoustic processing unit;

[0060] Based on the tuned secondary prosody representation correction module and the tuned acoustic processing unit, a corresponding acoustic model is constructed.

[0061] In some feasible implementations, the acoustic model further includes at least an acoustic processing unit, and the device further includes:

[0062] A content acquisition module, used to acquire a target phoneme sequence, a target prosody label and a target speaker identifier, wherein the target prosody label at least includes a predicted primary prosody label, a predicted secondary prosody label and a predicted tertiary prosody label;

[0063] An input module, used for inputting the target phoneme sequence, the target speaker identifier, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the secondary prosody representation correction module to correct the prosody label and obtain a corrected secondary prosody label;

[0064] A combining module, for combining the target phoneme sequence, the target speaker identifier, the predicted primary prosody label, the corrected secondary prosody label, and the predicted tertiary prosody label to obtain a target content representation;

[0065] A synthesis module is used to input the target content representation into the acoustic processing unit for audio synthesis to obtain target audio corresponding to the target factor sequence.

[0066] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0067] The memory is used to store computer programs;

[0068] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0069] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0070] The embodiments of the present invention include the following advantages:

[0071] In an embodiment of the present invention, by obtaining training data for acoustic model training, the training data includes at least text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, and the acoustic model includes at least a secondary prosodic representation correction module. The text content, audio pair data, acoustic features, and text prosodic features are first input into the secondary prosodic representation correction module to correct the prosodic label. Based on the correction result, the secondary prosodic representation correction module is parameter-tuned to obtain the optimized secondary prosodic representation correction module. Then, the model is performed according to the text content, audio pair data, acoustic features, and the optimized secondary prosodic representation correction module. Type training is performed to obtain at least two loss functions, and then the parameters of the acoustic processing unit are tuned according to the at least two loss functions to obtain an acoustic model that meets the preset conditions. The acoustic model is used to synthesize the target audio corresponding to the corresponding speaker, so that by constructing the corresponding training data, the same text prosodic features are used in the training process of the acoustic model, that is, the same prosodic distribution is used for model training, which provides the possibility for the correction of the prosodic distribution. At the same time, the model is optimized and a secondary prosodic representation correction module is constructed to support prosodic correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosodic style of different users while ensuring accurate correspondence with the real audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a schematic diagram of the rhythmic hierarchy relationship provided in an embodiment of the present invention;

[0073] Figure 2 is a flowchart of a method for generating an acoustic model provided in an embodiment of the present invention;

[0074] Figure 3 is a schematic diagram of a data sample provided in an embodiment of the present invention;

[0075] Figure 4 is a schematic diagram of the structure of an acoustic model provided in an embodiment of the present invention;

[0076] Figure 5 It is a structural block diagram of a device for generating an acoustic model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, some technical features involved in the embodiments of the present invention are explained and illustrated below:

[0079] Prosodic structure prediction: In phonetics, the prosodic structure can be divided into four levels according to the characteristics of speech, such as rhythm, stress, and intonation. Chinese phonetics research generally adopts four levels of prosodic structure: prosodic words, prosodic phrases, intonation phrases, and sentences. For example, Figure 1 , shows a schematic diagram of the rhythmic hierarchy relationship provided in an embodiment of the present invention, showing the rhythmic hierarchy relationship of a sentence of audio corresponding to the text, and using digital labels (#1, #2, #3, #4) for simplified representation, that is, "This pair of #1 shoes #2 is not strong #3, don't #1 wear it #4.". As can be seen from the figure, the division of rhythmic words is very close to the grammatical participle structure, the boundaries of intonation phrases mostly coincide with sentence-breaking punctuation marks such as "commas", and the sentence level always coincides with the sentence-end punctuation marks. Among them, rhythmic structure prediction is the process of predicting the rhythmic digital labels of audio texts through methods such as rules or neural networks. In practice, the rhythmic structure prediction is generally performed on single-sentence texts, and the sentence (#4) label will appear only at the end of the sentence, so #4 does not need to be predicted. In the prediction results of the other three levels of prosodic structure, the accuracy of prosodic words (#1) and intonation phrases (#3) is usually very high, with the harmonic mean reaching more than 95%, while the harmonic mean of prosodic phrases (#2) is generally around 75% to 85%. This is because prosodic phrases are highly subjective, and different speakers may have different ways of dividing prosodic phrases when speaking the same text, and there is often no single correct division.

[0080] As an example, with the development of large speech models, the scaling law reveals that as the amount of training data increases, the sound quality and naturalness of the synthesized audio will be greatly improved. This encourages us to use as much audio data without prosody annotation as possible at a low cost, but it also introduces new problems. If the acoustic model is not trained with explicit prosody structure information at all, the correctness of the semantic pauses of the output audio cannot be guaranteed, and even the division of intonation phrases (#3) will be wrong, which greatly affects understanding. Even large speech synthesis models (such as Vall-E, Cosyvoice, etc.) trained with a large amount of audio data will have frequent semantic pause errors. For acoustic models modeled with explicit prosody information, here we consider the use of two major types of sound library data to analyze the problem. The first category, "prosody precise annotation" sound library data, refers to the sound library data with complete prosody structure labels annotated based on the correspondence between reference text and audio. The second category, "no prosody annotation", refers to the sound library data with only text corresponding to the audio and no prosody structure annotations. Among them, the latter is generally automatically annotated through speech recognition. It is assumed here that this type of sound library can ensure that the sentence punctuation of the text is aligned with the audio, that is, where there are sentence punctuation (such as periods, exclamation marks, question marks, etc.) and commas, the corresponding audio must have long silent paragraphs or obvious pauses. When using the second type of data, a separately trained prosody prediction model is usually used to pre-annotate the text with prosody labels, and then the acoustic model is trained. However, in the process of building the corresponding acoustic model, there is a problem that the different prosodic styles of different speakers cannot be reflected, and the accurate correspondence with the real audio cannot be guaranteed.

[0081] In this regard, in the present invention, by obtaining training data for acoustic model training, the training data includes at least text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, and the acoustic model includes at least a secondary prosodic representation correction module. The text content, audio pair data, acoustic features, and text prosodic features are first input into the secondary prosodic representation correction module to correct the prosodic label, and the secondary prosodic representation correction module is parameter-optimized based on the correction result to obtain the optimized secondary prosodic representation correction module. Then, the model is performed according to the text content, audio pair data, acoustic features, and the optimized secondary prosodic representation correction module. Type training is performed to obtain at least two loss functions, and then the parameters of the acoustic processing unit are tuned according to the at least two loss functions to obtain an acoustic model that meets the preset conditions. The acoustic model is used to synthesize the target audio corresponding to the corresponding speaker, so that by constructing the corresponding training data, the same text prosodic features are used in the training process of the acoustic model, that is, the same prosodic distribution is used for model training, which provides the possibility for the correction of the prosodic distribution. At the same time, the model is optimized and a secondary prosodic representation correction module is constructed to support prosodic correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosodic style of different users while ensuring accurate correspondence with the real audio.

[0082] Reference Figure 2 , shows a flowchart of a method for generating an acoustic model provided in an embodiment of the present invention, which may specifically include the following steps:

[0083] Step 201, obtaining training data for acoustic model training, wherein the training data at least includes text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, and the acoustic model at least includes a secondary prosodic representation correction module and an acoustic processing unit;

[0084] In an embodiment of the present invention, training data corresponding to an acoustic model based on a speaker-corrected prosodic label can be first constructed, and the acoustic model can be trained by obtaining data such as text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data as corresponding training data. Among them, text content refers to natural language text input to the acoustic model, which can be a sentence, paragraph, or entire document, etc.; audio pair data refers to speech audio data corresponding to specific text content, etc., and the present invention does not limit this.

[0085] In the specific implementation, for each text and audio pair data collected, it is necessary to provide necessary acoustic features (speaker ID, Mel spectrum, fundamental frequency, energy, etc.) and text prosody features. Furthermore, in order to ensure the consistency of prosody distribution during training and inference, the prosody labels predicted by the prosody model are uniformly used as model input in the embodiment of the present invention. At the same time, in order to ensure the correspondence between prosody labels and audio in training as much as possible, each prosody level label is processed separately:

[0086] For the #1 label, it is believed that the prediction accuracy is high, so the predicted result can be used directly; for the #3 label, it is necessary to ensure that the longer silent segments in the training audio correspond to the predicted #3, which can be achieved by marking the position of the longer silent segments in the audio with commas, and it is assumed that the rhythm prediction model can accurately predict the rhythm at the comma as #3; but for the #2 label, which is highly subjective and has a low prediction accuracy, this patent introduces a modified #2 to guide model training and reasoning. Therefore, in the input part, not only the predicted #2 but also some real #2 labels need to be given.

[0087] The input content corresponding to the specific text prosodic features is as follows:

[0088] "Rhythm precision labeling" sound library: phoneme sequence, predicted first-level prosody label ID, predicted second-level prosody label ID, predicted third-level prosody label ID, and real second-level prosody label ID.

[0089] "No prosody annotation" sound library: phoneme sequence, predicted first-level prosody label ID, predicted second-level prosody label ID, predicted third-level prosody label ID (optional: real second-level prosody label ID).

[0090] Based on the differences in text prosodic features, the training data can be divided into two groups. The first group can be labeled data containing real secondary prosodic label IDs, and the second group can be unlabeled data that does not contain real secondary prosodic label IDs. The acoustic model is trained based on the labeled data and the unlabeled data, so that for the "prosodic precise labeling" sound library data, the same prosodic distribution is used as input during acoustic model training and reasoning. For the "unlabeled prosodic labeling" sound library data, the gap between the input prosodic label and the original audio prosody is reduced when participating in the acoustic model training, thereby improving the prediction accuracy.

[0091] In addition, for the "unrhythmic annotation" sound library, a small number of audio-text pairs can be selected for manual annotation of real #2 labels, and semi-supervised training can be performed in subsequent experiments.

[0092] In one example, referring to Figure 3, shows a schematic diagram of a data sample provided in an embodiment of the present invention, wherein sil of the phoneme sequence represents the start and end silence segments, sp3 represents the long silence segment corresponding to #3 in the sentence, ^ represents the zero initial consonant phoneme; 0 of the rhythm tag ID represents a non-boundary, and 1 represents a boundary.

[0093] It should be noted that a phoneme sequence refers to a group of phonemes used to represent the pronunciation of a specific word, phrase or sentence in a language. A phoneme is the smallest unit in phonetics that can distinguish meanings, that is, two different phonemes may correspond to two different meanings in the same language.

[0094] For example, in English, the word "cat" can be broken down into three phonemes: / k / , Similarly, the word "bat" can be broken down into three phonemes: / b / , / t ... and / t / . Although both words have a and a / t / , but because the first phoneme is different ( / k / and / b / ), the two words have different meanings.

[0095] After the corresponding training data is constructed, the acoustic model to be trained can be obtained, and the acoustic model can be used to synthesize the target audio corresponding to the corresponding speaker. Optionally, the acoustic model can include an input layer, a secondary prosody representation correction module, and an acoustic processing unit. The input layer can be used to input the corresponding training data, and the secondary prosody representation correction module can be used to correct the corresponding prosody labels to alleviate the problem of inconsistency between text prosody features and audio during unlabeled data training; the acoustic processing unit can be used to encode, predict, decode, etc. the mapped vector features to synthesize the corresponding target audio.

[0096] Step 202: input the text content, the audio pair data, the acoustic features, and the text prosody features into the secondary prosody representation correction module to correct the prosody label, and optimize the parameters of the secondary prosody representation correction module based on the correction result to obtain an optimized secondary prosody representation correction module;

[0097] During the training process of the acoustic model, the parameters of the secondary prosody representation correction module can be optimized first. Specifically, the text content, audio pair data, acoustic features and text prosody features can be input into the secondary prosody representation correction module to correct the prosody labels. Based on the correction result, the parameters of the secondary prosody representation correction module can be optimized to obtain the optimized secondary prosody representation correction module. The secondary prosody labels in the prediction process can be corrected through the secondary prosody representation correction module, thereby alleviating the problem of inconsistency between text prosody features and audio, and ensuring accurate correspondence with the real audio.

[0098] It should be noted that, since the training data includes labeled data and unlabeled data, the labeled data contains the real secondary prosody label, which can be compared with the secondary prosody label predicted by the secondary prosody representation correction module during the training process, and the corresponding prosody label prediction loss can be calculated. Then, the parameters of the secondary prosody representation correction module are tuned based on the prosody label prediction loss, so as to optimize the model parameters corresponding to the secondary prosody representation correction module; and for the unlabeled data, since it does not contain the real secondary prosody label, when the secondary prosody representation correction module is trained with the unlabeled data, the obtained prosody label prediction loss is 0, that is, during the training process, the model of the secondary prosody representation correction module can be tuned only based on the labeled data.

[0099] In a specific implementation, the text prosody features include at least a first-class prosody label, and the first-class prosody label includes at least a predicted first-class prosody label, a predicted second-class prosody label, a predicted third-class prosody label, and a real second-class prosody label. Then, the text content, audio pair data, acoustic features, predicted first-class prosody labels, predicted second-class prosody labels, and predicted third-class prosody labels can be first input into a second-class prosody representation correction module for prediction to obtain a corresponding prediction result. Then, the prediction result is compared with the real second-class prosody label to determine the prosody label prediction loss for the second-class prosody representation correction module. Then, the prosody label prediction loss is used to perform parameter tuning on the second-class prosody representation correction module until the prosody label prediction loss meets the preset conditions to obtain the optimized second-class prosody representation correction module.

[0100] Optionally, the acoustic features include at least a speaker identification, and the feature input process can also be: obtaining the phoneme features corresponding to the audio pair data, and then inputting the text content, speaker identification, phoneme features, predicted first-level prosody labels, predicted second-level prosody labels, and predicted third-level prosody labels into the second-level prosody representation correction module for prediction, obtaining the corresponding prediction results, and then executing the model tuning process in the above embodiment, thereby alleviating the problem of inconsistency between the text prosody features and the audio, and ensuring accurate correspondence with the real audio.

[0101] It should be noted that in the embodiment of the present invention, an acoustic model of a FastPitch-like structure can be used for modeling, and the encoder part thereof is modified. Specifically, the encoder mainly adds prosodic features, and before the phoneme representation is added to the prosodic representation of each level, a secondary prosodic label corrector module (i.e., a secondary prosodic representation correction module) is introduced, the predicted secondary prosodic phrase ID is first corrected, and then the corrected secondary prosodic phrase ID is used to obtain the final secondary prosodic representation. Among them, the key component involved in the acoustic model is the secondary prosodic label corrector (i.e., the secondary prosodic representation correction module), the input conditions include text content, speaker information, and predicted primary, secondary, and tertiary prosodic label IDs, and the output is the secondary prosodic label ID corrected by the speaker on the corresponding text, and the model structure is the same as the duration predictor of FastPitch, i.e., it is composed of two layers of one-dimensional convolutional structures.

[0102] In one example, referring to Figure 4 , showing a schematic diagram of the structure of the acoustic model provided in an embodiment of the present invention, for the model tuning process of the secondary prosody representation correction module (i.e., the secondary prosody representation correction module), the speaker ID can be firstly feature extracted to obtain the corresponding speaker representation (such as extracting the acoustic features corresponding to the speaker from the audio, etc.), and the phoneme ID can be feature extracted to obtain the corresponding phoneme representation (such as extracting the corresponding phoneme features from the text content, etc.), and then the predicted primary prosody label ID, the predicted secondary prosody label ID, and the predicted tertiary prosody label ID are combined and input into the secondary prosody representation correction module for prediction to obtain the predicted After the secondary prosody label ID is obtained, the predicted secondary prosody label ID is compared with the actual secondary prosody label ID, the secondary prosody label prediction loss is calculated, and then the secondary prosody representation correction module is reversely tuned based on the secondary prosody label prediction loss until the secondary prosody label prediction loss is less than or equal to the preset threshold. Then, the secondary prosody representation correction module is stopped to obtain the tuned secondary prosody representation correction module, thereby supporting prosody correction based on the characteristics of the speaker by introducing secondary prosody label correction, which helps to alleviate the problem of inconsistency between text prosody features and audio and ensures accurate correspondence with real audio.

[0103] Step 203, performing model training according to the text content, the audio pair data, the acoustic features, and the tuned secondary prosody representation correction module to obtain at least two loss functions;

[0104] After completing the model tuning of the secondary prosody representation correction module, model training can be further performed based on the training data and the tuned secondary prosody representation correction module to obtain at least two corresponding loss functions, and then the acoustic processing unit in the acoustic model is model tuned based on the at least two loss functions to obtain the corresponding acoustic model.

[0105] In an embodiment of the present invention, the acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor. The encoder can encode the input features to obtain a high-dimensional representation, and then input the high-dimensional representation to the variable feature predictor to predict the variable features of the audio, such as pitch, energy, etc., and input the variable features to the decoder for decoding, so as to decode the content representation, rhythm representation, and variable features into corresponding Mel spectra, thereby synthesizing the corresponding audio.

[0106] For labeled data, the text content, audio pair data, acoustic features, predicted first-level prosody labels, predicted second-level prosody labels and predicted third-level prosody labels can be first input into the tuned second-level prosody representation correction module to obtain the corrected second-level prosody labels, and then the text content, audio pair data, acoustic features, predicted first-level prosody labels, corrected second-level prosody labels and predicted third-level prosody labels are used to represent and combine to obtain the corresponding first content representation, and then the first content representation is input into the encoder for encoding processing to obtain the corresponding first intermediate representation, and then the first intermediate representation is input into the variable feature predictor for prediction to obtain the first variable feature related to the speech, and the variable feature is compared with the preset first real content, and the corresponding first variable feature prediction loss is calculated, and then the variable feature is input into the decoder for decoding processing to obtain the corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with the preset second real content, and the corresponding first Mel spectrum prediction loss is calculated.

[0107] For unlabeled data, the text prosodic features also include at least a second type of prosodic label, and the second type of prosodic label includes at least a predicted first-level prosodic label, a predicted second-level prosodic label, and a predicted third-level prosodic label. The corresponding training process can firstly adopt the text content, audio pair data, acoustic features, predicted first-level prosodic labels, predicted second-level prosodic labels, and predicted third-level prosodic labels for representation and combination to obtain the corresponding second content representation, and input the second content representation into the encoder for encoding processing to obtain the corresponding second intermediate representation, and then input the second intermediate representation into the variable feature predictor for prediction to obtain the second variable feature related to the speech, and compare the variable feature with the preset first real content, calculate the corresponding second variable feature prediction loss, and then input the variable feature into the decoder for decoding processing to obtain the corresponding predicted Mel spectrum, and compare the predicted Mel spectrum with the preset second real content, and calculate the corresponding second Mel spectrum prediction loss.

[0108] It should be noted that when the data without manually annotated secondary prosody labels is input into the model, the parameters are only updated through the mel spectrum prediction loss and variable feature prediction loss, which has an indirect impact on the secondary prosody label corrector, that is, the corrected secondary prosody label should help reduce the mel spectrum prediction loss and help predict the variable features of the current audio. By providing the possibility of modifying the secondary prosody label, the inconsistency problem between the prosody features of the text and the audio during unlabeled data training is alleviated.

[0109] Step 204, tuning parameters of the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions, wherein the acoustic model is used to synthesize target audio corresponding to the corresponding speaker.

[0110] After obtaining loss functions such as variable feature prediction loss and Mel spectrum prediction loss through the above process, the parameters of the acoustic processing unit can be comprehensively tuned according to at least two loss functions obtained, and the parameter tuning can be stopped when the loss functions all meet the stopping iteration conditions to obtain the tuned acoustic processing unit, thereby constructing a corresponding acoustic model based on the tuned secondary prosody representation correction module and the tuned acoustic processing unit, thereby constructing corresponding training data and using the same text prosody features in the training process of the acoustic model, that is, using the same prosody distribution for model training, which provides the possibility for correcting the prosody distribution, and at the same time optimizes the model and constructs a secondary prosody representation correction module to support prosody correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosody styles of different users while ensuring accurate correspondence with the real audio.

[0111] In a specific implementation, based on the obtained loss function, the parameters of the acoustic processing unit can be tuned according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss and the second Mel spectrum prediction loss until the iteration condition is met to obtain the tuned acoustic processing unit, and then based on the tuned secondary prosody representation correction module and the tuned acoustic processing unit, the corresponding acoustic model is constructed, so that by constructing corresponding training data, the same text prosody features are used in the training process of the acoustic model, that is, the same prosody distribution is used for model training, which provides the possibility for the correction of the prosody distribution, and at the same time, the model is optimized, and a secondary prosody representation correction module is constructed to support prosody correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosody styles of different users, while ensuring accurate correspondence with the real audio.

[0112] It should be noted that for variable feature prediction loss, Mel spectrum prediction loss, etc., corresponding thresholds can be set respectively. When the loss function is less than or equal to the preset threshold, the model iteration can be stopped to complete the model tuning process. For example, a first threshold for variable feature prediction loss and a second threshold for Mel spectrum prediction loss can be set. Regardless of whether the model is trained with labeled data or with unlabeled data, when the variable feature prediction loss is less than or equal to the first threshold and the Mel spectrum prediction loss is less than or equal to the second threshold, the parameter tuning of the acoustic processing unit can be stopped to complete the model tuning process, so as to construct the corresponding acoustic model in combination with the tuned secondary rhythm representation correction module.

[0113] Based on the above process, after the corresponding acoustic model is trained, audio synthesis can be performed through the acoustic model. Specifically, the target phoneme sequence, target prosody label and target speaker identification can be obtained, and the target prosody label includes at least a predicted primary prosody label, a predicted secondary prosody label and a predicted tertiary prosody label. Then, the target phoneme sequence, the target speaker identification and the predicted primary prosody label, the predicted secondary prosody label and the predicted tertiary prosody label are input into the secondary prosody representation correction module to correct the prosody label and obtain the corrected secondary prosody label. Then, the target phoneme sequence, the target speaker identification and the predicted primary prosody label, the predicted secondary prosody label and the corrected tertiary prosody label are input into the secondary prosody representation correction module to correct the prosody label and obtain the corrected secondary prosody label. The corrected secondary prosody label and the predicted tertiary prosody label are combined to obtain the target content representation, which is then input into the acoustic processing unit for audio synthesis to obtain the target audio corresponding to the target factor sequence, so as to obtain the secondary prosody pause that conforms to the speaker's style after correction by the secondary prosody label corrector. Finally, the model synthesizes the Mel spectrum that conforms to the corresponding rhythm, and the vocoder converts the corresponding audio. At the same time, for the same input text, it can not only obtain more accurate primary prosody labels and tertiary prosody labels based on the general prosody model, but also fine-tune the secondary prosody labels according to the prosodic characteristics of different speakers to achieve intra-sentence pauses, so that the synthesis result is more in line with the speaker's style.

[0114] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0115] In an embodiment of the present invention, by obtaining training data for acoustic model training, the training data includes at least text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, and the acoustic model includes at least a secondary prosodic representation correction module. The text content, audio pair data, acoustic features, and text prosodic features are first input into the secondary prosodic representation correction module to correct the prosodic label. Based on the correction result, the secondary prosodic representation correction module is parameter-tuned to obtain the optimized secondary prosodic representation correction module. Then, the model is performed according to the text content, audio pair data, acoustic features, and the optimized secondary prosodic representation correction module. Type training is performed to obtain at least two loss functions, and then the parameters of the acoustic processing unit are tuned according to the at least two loss functions to obtain an acoustic model that meets the preset conditions. The acoustic model is used to synthesize the target audio corresponding to the corresponding speaker, so that by constructing the corresponding training data, the same text prosodic features are used in the training process of the acoustic model, that is, the same prosodic distribution is used for model training, which provides the possibility for the correction of the prosodic distribution. At the same time, the model is optimized and a secondary prosodic representation correction module is constructed to support prosodic correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosodic style of different users while ensuring accurate correspondence with the real audio.

[0116] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0117] Reference Figure 4 The acoustic model shown in the figure, the corresponding model building process may include:

[0118] S01. Construct training data for an acoustic model based on speaker-corrected prosodic labels.

[0119] For each piece of text and audio pair data collected, it is necessary to provide necessary acoustic features (speaker ID, Mel spectrum, fundamental frequency, energy, etc.) and text prosodic features.

[0120] In order to ensure the consistency of prosody distribution during training and inference, this patent uniformly uses the prosody labels predicted by the prosody model as model input. In order to ensure the correspondence between prosody labels and audio in training as much as possible, each prosody level label is processed separately:

[0121] For the #1 label, it is considered that the prediction accuracy is high, so the predicted result can be used directly.

[0122] For the #3 label, it is necessary to ensure that the longer silent segments in the training audio correspond to the predicted #3. This can be achieved by using commas to mark the positions of the longer silent segments in the audio. It is assumed here that the rhythm prediction model can accurately predict the rhythm at the comma as #3.

[0123] However, for the #2 label, which is highly subjective and has low prediction accuracy, this patent introduces a modified #2 to guide model training and reasoning. Therefore, in the input part, not only the predicted #2 but also some real #2 labels need to be given. The specific text rhythm input content is as follows:

[0124] "Rhythm precision labeling" sound library: phoneme sequence, predicted first-level prosody label ID, predicted second-level prosody label ID, predicted third-level prosody label ID, and real second-level prosody label ID.

[0125] "No prosody annotation" sound library: phoneme sequence, predicted first-level prosody label ID, predicted second-level prosody label ID, predicted third-level prosody label ID (optional: real second-level prosody label ID).

[0126] Among them, the "unrhythmic annotation category" sound library can select a small number of audio text pairs for manual annotation of real #2 labels, and conduct semi-supervised training in subsequent experiments.

[0127] Among them, sil in the phoneme sequence represents the start and end silence segments, sp3 represents the long silence segment corresponding to #3 in the sentence, ^ represents the zero initial consonant phoneme; 0 in the prosody tag ID represents a non-boundary, and 1 represents a boundary.

[0128] S02. Construct an acoustic model based on speaker-corrected prosodic labels.

[0129] This patent uses an acoustic model similar to the FastPitch structure for modeling and modifies its encoder part. The main changes to the encoder are the addition of prosodic representation and the introduction of a secondary prosodic label corrector module before adding the phoneme representation to the prosodic representations at each level. The predicted secondary prosodic phrase ID is first corrected, and then the corrected secondary prosodic phrase ID is used to obtain the final secondary prosodic representation.

[0130] The key component is the secondary prosody label corrector, whose input conditions include text content, speaker information, and predicted primary, secondary, and tertiary prosody label IDs. The output is the secondary prosody label ID corrected by the speaker on the corresponding text. The model structure is the same as the duration predictor of FastPitch, that is, it consists of two layers of one-dimensional convolutional structures. Figure 3 】as shown.

[0131] S03. Add semi-supervised secondary prosody label prediction loss and train model parameters.

[0132] Based on the original FastPitch model training loss (Mel spectrum prediction loss, variable feature prediction loss), the cross entropy loss of the secondary prosody label prediction is combined to jointly guide the training of the constructed acoustic model. Among them, the secondary prosody label prediction loss is mainly used to directly guide the learning of the secondary prosody label corrector.

[0133] For most of the "unlabeled" sound libraries, there is no data with manually labeled secondary prosody labels, and the corresponding secondary prosody label prediction loss is 0, while the corresponding loss of other data is greater than 0. When the data without manually labeled secondary prosody labels is input into the model, the parameters are updated only through the Mel spectrum prediction loss and the variable feature prediction loss, which has an indirect impact on the secondary prosody label corrector, that is, the corrected secondary prosody label should help reduce the Mel spectrum prediction loss and help predict the variable features of the current audio.

[0134] By providing the possibility of modifying the secondary prosodic labels, the inconsistency problem between the prosodic features of text and audio during training with unlabeled data is alleviated.

[0135] S04. Reasoning supports the synthesis of different prosody by different speakers.

[0136] Using the acoustic model trained above, the phoneme sequence ID and the predicted first, second, and third level prosody label IDs and speaker ID are input, and the second level prosody pauses that match the speaker's style are obtained after correction by the second level prosody label corrector. Finally, the model synthesizes the Mel spectrum that matches the corresponding prosody, and the vocoder converts the corresponding audio. It can be seen that for the same input text, it is possible to obtain more accurate #1 and #3 based on the general prosody model, and fine-tune the #2 intra-sentence pauses according to the prosody characteristics of different speakers, so that the synthesis result is more in line with the speaker's style.

[0137] In the above process, a unique rhythm feature of acoustic model training data is constructed, which uses both the same hierarchical rhythm prediction labels as those distributed during inference and optional manually labeled secondary rhythm labels. This makes it possible to correct the rhythm distribution in real time based on information such as content and speaker. At the same time, in the FastPitch-like acoustic model structure, a secondary rhythm label corrector is introduced to support rhythm correction based on speaker characteristics. In addition, semi-supervised learning that supports secondary rhythm label modification indirectly promotes the modified rhythm label to be close to the original audio rhythm by reducing alignment loss and Mel loss.

[0138] Based on the above process, the technical solution involved in the embodiment of the present invention can at least achieve the following beneficial effects:

[0139] For the "rhythm-precisely labeled" sound library data, the same rhythm distribution is used as input during acoustic model training and inference; for the "unrhythm-annotated" sound library data, the gap between the input rhythm labels and the original audio rhythm is reduced when participating in acoustic model training; when inferring the acoustic model, it supports the synthesis of unique and stable rhythm effects based on the characteristics of the speaker.

[0140] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0141] Reference Figure 5 , shows a structural block diagram of a device for generating an acoustic model provided in an embodiment of the present invention, which may specifically include the following modules:

[0142] A data acquisition module 501 is used to acquire training data for acoustic model training, wherein the training data at least includes text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, and the acoustic model at least includes a secondary prosodic representation correction module and an acoustic processing unit;

[0143] A correction module 502 is used to input the text content, the audio pair data, the acoustic features and the text prosody features into the secondary prosody representation correction module to correct the prosody label, and optimize the parameters of the secondary prosody representation correction module based on the correction result to obtain an optimized secondary prosody representation correction module;

[0144] A training module 503 is used to perform model training according to the text content, the audio pair data, the acoustic features and the tuned secondary prosody representation correction module to obtain at least two loss functions;

[0145] The tuning module 504 is used to perform parameter tuning on the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions, and the acoustic model is used to synthesize the target audio corresponding to the corresponding speaker.

[0146] In some feasible implementations, the text prosodic features include at least a first type of prosodic label, the first type of prosodic label includes at least a predicted first level prosodic label, a predicted second level prosodic label, a predicted third level prosodic label, and a real second level prosodic label, and the correction module 502 is specifically used to:

[0147] Inputting the text content, the audio pair data, the acoustic features, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the secondary prosody representation correction module for prediction to obtain a corresponding prediction result;

[0148] Comparing the prediction result with the true secondary prosody label to determine the prosody label prediction loss for the secondary prosody representation correction module;

[0149] The prosody label prediction loss is used to perform parameter tuning on the secondary prosody representation correction module until the prosody label prediction loss meets a preset condition, thereby obtaining a tuned secondary prosody representation correction module.

[0150] In some feasible implementations, the acoustic feature includes at least a speaker identifier, and the correction module 502 is specifically configured to:

[0151] Obtaining phoneme features corresponding to the audio pair data;

[0152] The text content, the speaker identifier, the phoneme feature, the predicted primary prosody label, the predicted secondary prosody label and the predicted tertiary prosody label are input into the secondary prosody representation correction module for prediction to obtain corresponding prediction results.

[0153] In some feasible implementations, the acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor, and the training module 503 is specifically used to:

[0154] Inputting the text content, the audio pair data, the acoustic features, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the optimized secondary prosody representation correction module to obtain a corrected secondary prosody label;

[0155] The text content, the audio pair data, the acoustic features, the predicted primary prosody label, the corrected secondary prosody label, and the predicted tertiary prosody label are used to perform representation combination to obtain a corresponding first content representation;

[0156] Inputting the first content representation into the encoder for encoding to obtain a corresponding first intermediate representation;

[0157] Inputting the first intermediate representation into the variable feature predictor for prediction, obtaining a first variable feature related to speech, and comparing the variable feature with a preset first real content, and calculating a corresponding first variable feature prediction loss;

[0158] The variable feature is input into the decoder for decoding processing to obtain a corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with a preset second real content to calculate the corresponding first Mel spectrum prediction loss.

[0159] In some feasible implementations, the text prosodic features include at least a second type of prosodic label, the second type of prosodic label includes at least a predicted primary prosodic label, a predicted secondary prosodic label, and a predicted tertiary prosodic label, and the training module 503 is specifically used for:

[0160] The text content, the audio pair data, the acoustic feature, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label are used to perform representation combination to obtain a corresponding second content representation;

[0161] Inputting the second content representation into the encoder for encoding to obtain a corresponding second intermediate representation;

[0162] Inputting the second intermediate representation into the variable feature predictor for prediction, obtaining a second variable feature related to speech, and comparing the variable feature with a preset first real content, and calculating a corresponding second variable feature prediction loss;

[0163] The variable feature is input into the decoder for decoding processing to obtain a corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with a preset second real content to calculate a corresponding second Mel spectrum prediction loss.

[0164] In some feasible implementations, the tuning module 504 is specifically used to:

[0165] Optimizing parameters of the acoustic processing unit according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until an iteration condition is met, thereby obtaining an optimized acoustic processing unit;

[0166] Based on the tuned secondary prosody representation correction module and the tuned acoustic processing unit, a corresponding acoustic model is constructed.

[0167] In some feasible implementations, the acoustic model further includes at least an acoustic processing unit, and the device further includes:

[0168] A content acquisition module, used to acquire a target phoneme sequence, a target prosody label and a target speaker identifier, wherein the target prosody label at least includes a predicted primary prosody label, a predicted secondary prosody label and a predicted tertiary prosody label;

[0169] An input module, used for inputting the target phoneme sequence, the target speaker identifier, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the secondary prosody representation correction module to correct the prosody label and obtain a corrected secondary prosody label;

[0170] A combining module, for combining the target phoneme sequence, the target speaker identifier, the predicted primary prosody label, the corrected secondary prosody label, and the predicted tertiary prosody label to obtain a target content representation;

[0171] A synthesis module is used to input the target content representation into the acoustic processing unit for audio synthesis to obtain target audio corresponding to the target factor sequence.

[0172] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] In addition, an embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned acoustic model generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0174] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned acoustic model generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0175] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0176] It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program codes.

[0177] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0180] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0181] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0182] The above is a detailed introduction to a method for generating an acoustic model and a device for generating an acoustic model provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for generating an acoustic model, characterized in that: include: Acquire training data for acoustic model training, the training data at least including text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, the acoustic model at least including a secondary prosodic representation correction module and an acoustic processing unit; Inputting the text content, the audio pair data, the acoustic features and the text prosody features into the secondary prosody representation correction module to correct the prosody label, and optimizing the parameters of the secondary prosody representation correction module based on the correction result to obtain an optimized secondary prosody representation correction module; Performing model training according to the text content, the audio pair data, the acoustic features, and the tuned secondary prosody representation correction module to obtain at least two loss functions; The acoustic processing unit is parameter-tuned according to at least two of the loss functions to obtain an acoustic model that meets preset conditions, and the acoustic model is used to synthesize the target audio corresponding to the corresponding speaker.

2. The method according to claim 1, characterized in that The text prosody features include at least a first-class prosody label, and the first-class prosody labels include at least a predicted first-class prosody label, a predicted second-class prosody label, a predicted third-class prosody label, and a real second-class prosody label. The text content, the audio pair data, the acoustic features, and the text prosody features are input into the second-class prosody representation correction module to correct the prosody labels, and the second-class prosody representation correction module is parameter-tuned based on the correction result to obtain the tuned second-class prosody representation correction module, including: Inputting the text content, the audio pair data, the acoustic features, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the secondary prosody representation correction module for prediction to obtain a corresponding prediction result; Comparing the prediction result with the true secondary prosody label to determine the prosody label prediction loss for the secondary prosody representation correction module; The prosody label prediction loss is used to perform parameter tuning on the secondary prosody representation correction module until the prosody label prediction loss meets a preset condition, thereby obtaining a tuned secondary prosody representation correction module.

3. The method according to claim 2, characterized in that The acoustic feature at least includes a speaker identifier, and the text content, the audio pair data, the acoustic feature, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label are input into the secondary prosody representation correction module for prediction to obtain a corresponding prediction result, including: Obtaining phoneme features corresponding to the audio pair data; The text content, the speaker identifier, the phoneme feature, the predicted primary prosody label, the predicted secondary prosody label and the predicted tertiary prosody label are input into the secondary prosody representation correction module for prediction to obtain corresponding prediction results.

4. The method according to claim 2 or 3, characterized in that: The acoustic processing unit at least includes an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor. The model training is performed according to the text content, the audio pair data, the acoustic features, and the tuned secondary prosody representation correction module to obtain at least two loss functions, including: Inputting the text content, the audio pair data, the acoustic features, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the optimized secondary prosody representation correction module to obtain a corrected secondary prosody label; The text content, the audio pair data, the acoustic features, the predicted primary prosody label, the corrected secondary prosody label, and the predicted tertiary prosody label are used to perform representation combination to obtain a corresponding first content representation; Inputting the first content representation into the encoder for encoding to obtain a corresponding first intermediate representation; Inputting the first intermediate representation into the variable feature predictor for prediction, obtaining a first variable feature related to speech, and comparing the variable feature with a preset first real content, and calculating a corresponding first variable feature prediction loss; The variable feature is input into the decoder for decoding processing to obtain a corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with a preset second real content to calculate the corresponding first Mel spectrum prediction loss.

5. The method according to claim 4, characterized in that The text prosodic features include at least a second type of prosodic label, the second type of prosodic label includes at least a predicted primary prosodic label, a predicted secondary prosodic label, and a predicted tertiary prosodic label, and the model training is performed according to the text content, the audio pair data, the acoustic features, and the tuned secondary prosodic representation correction module to obtain at least two loss functions, including: The text content, the audio pair data, the acoustic feature, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label are used to perform representation combination to obtain a corresponding second content representation; Inputting the second content representation into the encoder for encoding to obtain a corresponding second intermediate representation; Inputting the second intermediate representation into the variable feature predictor for prediction, obtaining a second variable feature related to speech, and comparing the variable feature with a preset first real content, and calculating a corresponding second variable feature prediction loss; The variable feature is input into the decoder for decoding processing to obtain a corresponding predicted Mel spectrum, and the predicted Mel spectrum is compared with a preset second real content to calculate a corresponding second Mel spectrum prediction loss.

6. The method according to claim 5, characterized in that The step of tuning parameters of the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions includes: Optimizing parameters of the acoustic processing unit according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until an iteration condition is met, thereby obtaining an optimized acoustic processing unit; Based on the tuned secondary prosody representation correction module and the tuned acoustic processing unit, a corresponding acoustic model is constructed.

7. The method according to claim 1, characterized in that The acoustic model at least further comprises an acoustic processing unit, and the method further comprises: Acquire a target phoneme sequence, a target prosody label, and a target speaker identifier, wherein the target prosody label at least includes a predicted primary prosody label, a predicted secondary prosody label, and a predicted tertiary prosody label; Inputting the target phoneme sequence, the target speaker identifier, the predicted primary prosody label, the predicted secondary prosody label, and the predicted tertiary prosody label into the secondary prosody representation correction module to correct the prosody label, and obtain a corrected secondary prosody label; Combining the target phoneme sequence, the target speaker identifier, the predicted primary prosody label, the corrected secondary prosody label, and the predicted tertiary prosody label to obtain a target content representation; The target content representation is input into the acoustic processing unit for audio synthesis to obtain target audio corresponding to the target factor sequence.

8. A device for generating an acoustic model, characterized in that: include: A data acquisition module, used to acquire training data for acoustic model training, wherein the training data at least includes text content, audio pair data corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pair data, and the acoustic model at least includes a secondary prosodic representation correction module and an acoustic processing unit; A correction module, used for inputting the text content, the audio pair data, the acoustic features and the text prosody features into the secondary prosody representation correction module to correct the prosody label, and performing parameter tuning on the secondary prosody representation correction module based on the correction result to obtain a tuned secondary prosody representation correction module; A training module, used for performing model training according to the text content, the audio pair data, the acoustic features and the tuned secondary prosody representation correction module to obtain at least two loss functions; A tuning module is used to perform parameter tuning on the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions, and the acoustic model is used to synthesize the target audio corresponding to the corresponding speaker.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images