Acoustic model generation method, device, electronic device and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for generating an acoustic model, relating to the field of artificial intelligence technology. The method includes: acquiring training data, including text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs; the acoustic model includes a secondary prosodic representation correction module and an acoustic processing unit; inputting the text content, audio pairs, acoustic features, and text prosodic features into the secondary prosodic representation correction module for correction; optimizing the parameters of the secondary prosodic representation correction module based on the correction results to obtain an optimized secondary prosodic representation correction module; training the model based on the text content, audio pairs, acoustic features, and the optimized secondary prosodic representation correction module to obtain a loss function; and optimizing the parameters of the acoustic processing unit based on the loss function to obtain an acoustic model that meets preset conditions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for generating an acoustic model, an apparatus for generating an acoustic model, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Speech synthesis is a technology that converts text into speech. It typically involves several steps: front-end text processing, acoustic model prediction of acoustic features (such as Mel spectrum), and a vocoder to convert the acoustic features into speech. The acoustic model is a crucial step in modeling the content, timbre, and prosodic features of the synthesized audio. It usually requires input phoneme-level text features to specify content information, speaker information to specify the synthesized timbre, and prosodic structure labels to ensure controllable prosodic pauses. However, in constructing the corresponding acoustic model, there are problems such as the inability to represent the different prosodic styles of different speakers and the inability to guarantee an accurate correspondence with real audio. Summary of the Invention

[0003] The present invention provides a method, apparatus, electronic device, and computer-readable storage medium for generating acoustic models, in order to solve or partially solve the problems that the models constructed during the construction of acoustic models cannot reflect the different prosodic styles of different speakers and cannot guarantee an accurate correspondence with real audio.

[0004] This invention discloses a method for generating an acoustic model, comprising:

[0005] Acquire training data for training an acoustic model. The training data includes at least text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs. The acoustic model includes at least a two-level prosodic representation correction module and an acoustic processing unit.

[0006] The text content, the audio pair data, the acoustic features, and the text prosodic features are input into the secondary prosodic representation correction module to correct the prosodic labels. Based on the correction results, the parameters of the secondary prosodic representation correction module are optimized to obtain the optimized secondary prosodic representation correction module.

[0007] The model is trained based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions.

[0008] The acoustic processing unit is parameter-tuned according to at least two of the loss functions to obtain an acoustic model that meets preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker.

[0009] In some feasible implementations, the text prosodic features include at least a first type of prosodic label, which includes at least a predicted primary prosodic label, a predicted secondary prosodic label, a predicted tertiary prosodic label, and a true secondary prosodic label. The step of inputting the text content, the audio pair data, the acoustic features, and the text prosodic features into the secondary prosodic representation correction module for prosodic label correction, and then optimizing the parameters of the secondary prosodic representation correction module based on the correction results to obtain the optimized secondary prosodic representation correction module, includes:

[0010] The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prediction to obtain the corresponding prediction results.

[0011] The prediction results are compared with the actual secondary prosodic labels to determine the prosodic label prediction loss for the secondary prosodic representation correction module.

[0012] The prosodic label prediction loss is used to fine-tune the parameters of the secondary prosodic representation correction module until the prosodic label prediction loss meets the preset conditions, thus obtaining the fine-tuned secondary prosodic representation correction module.

[0013] In some feasible implementations, the acoustic features include at least a speaker identifier. The step of inputting the text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label into the secondary prosodic representation correction module for prediction, and obtaining the corresponding prediction result, includes:

[0014] Obtain the phoneme features corresponding to the audio pair data;

[0015] The text content, the speaker identifier, the phoneme features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prediction to obtain the corresponding prediction results.

[0016] In some feasible implementations, the acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor. The model is trained based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions, including:

[0017] The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the optimized secondary prosodic representation correction module to obtain the corrected secondary prosodic label.

[0018] The text content, the audio pair data, the acoustic features, the predicted first-level prosodic label, the corrected second-level prosodic label, and the predicted third-level prosodic label are combined to obtain the corresponding first content representation.

[0019] The first content representation is input into the encoder for encoding processing to obtain the corresponding first intermediate representation;

[0020] The first intermediate representation is input into the variable feature predictor for prediction to obtain the first variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding first variable feature prediction loss.

[0021] The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second real content to calculate the corresponding first Mel spectrum prediction loss.

[0022] In some feasible implementations, the text prosodic features include at least a second type of prosodic label, which includes at least a predicted primary prosodic label, a predicted secondary prosodic label, and a predicted tertiary prosodic label. The model is trained based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions, including:

[0023] The corresponding second content representation is obtained by combining the text content, the audio pair data, the acoustic features, the predicted first-level prosodic label, the predicted second-level prosodic label, and the predicted third-level prosodic label.

[0024] The second content representation is input into the encoder for encoding processing to obtain the corresponding second intermediate representation;

[0025] The second intermediate representation is input into the variable feature predictor for prediction to obtain the second variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding second variable feature prediction loss.

[0026] The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second true content to calculate the corresponding second Mel spectrum prediction loss.

[0027] In some feasible implementations, the step of fine-tuning the parameters of the acoustic processing unit based on at least two of the loss functions to obtain an acoustic model that meets preset conditions includes:

[0028] The acoustic processing unit is parameter-tuned according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until the iteration conditions are met, and the tuned acoustic processing unit is obtained.

[0029] Based on the optimized secondary prosodic representation correction module and the optimized acoustic processing unit, a corresponding acoustic model is constructed.

[0030] In some feasible implementations, the acoustic model further includes at least an acoustic processing unit, and the method further includes:

[0031] Obtain the target phoneme sequence, target prosodic label, and target speaker identifier, wherein the target prosodic label includes at least the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label;

[0032] The target phoneme sequence, the target speaker identifier, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prosodic label correction to obtain the corrected secondary prosodic label.

[0033] The target phoneme sequence, the target speaker identifier, the predicted primary prosodic label, the corrected secondary prosodic label, and the predicted tertiary prosodic label are combined to obtain the target content representation.

[0034] The target content representation is input into the acoustic processing unit for audio synthesis to obtain the target audio corresponding to the target factor sequence.

[0035] This invention also discloses an apparatus for generating an acoustic model, comprising:

[0036] The data acquisition module is used to acquire training data for training the acoustic model. The training data includes at least text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs. The acoustic model includes at least a two-level prosodic representation correction module and an acoustic processing unit.

[0037] The correction module is used to input the text content, the audio pair data, the acoustic features and the text prosodic features into the secondary prosodic representation correction module to correct the prosodic labels, and to optimize the parameters of the secondary prosodic representation correction module based on the correction results to obtain the optimized secondary prosodic representation correction module.

[0038] The training module is used to train the model based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions.

[0039] The tuning module is used to tune the parameters of the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker.

[0040] In some feasible implementations, the text prosodic features include at least a first type of prosodic label, which includes at least a predicted primary prosodic label, a predicted secondary prosodic label, a predicted tertiary prosodic label, and a true secondary prosodic label. The correction module is specifically used for:

[0041] The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prediction to obtain the corresponding prediction results.

[0042] The prediction results are compared with the actual secondary prosodic labels to determine the prosodic label prediction loss for the secondary prosodic representation correction module.

[0043] The prosodic label prediction loss is used to fine-tune the parameters of the secondary prosodic representation correction module until the prosodic label prediction loss meets the preset conditions, thus obtaining the fine-tuned secondary prosodic representation correction module.

[0044] In some feasible implementations, the acoustic features include at least a speaker identifier, and the correction module is specifically used for:

[0045] Obtain the phoneme features corresponding to the audio pair data;

[0046] The text content, the speaker identifier, the phoneme features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prediction to obtain the corresponding prediction results.

[0047] In some feasible implementations, the acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor, wherein the training module is specifically used for:

[0048] The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the optimized secondary prosodic representation correction module to obtain the corrected secondary prosodic label.

[0049] The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the corrected secondary prosodic label, and the predicted tertiary prosodic label are combined to obtain the corresponding first content representation.

[0050] The first content representation is input into the encoder for encoding processing to obtain the corresponding first intermediate representation;

[0051] The first intermediate representation is input into the variable feature predictor for prediction to obtain the first variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding first variable feature prediction loss.

[0052] The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second real content to calculate the corresponding first Mel spectrum prediction loss.

[0053] In some feasible implementations, the text prosodic features include at least a second type of prosodic label, which includes at least a predicted first-level prosodic label, a predicted second-level prosodic label, and a predicted third-level prosodic label. The training module is specifically used for:

[0054] The text content, the audio pair data, the acoustic features, the predicted first-level prosodic labels, the predicted second-level prosodic labels, and the predicted third-level prosodic labels are combined to obtain the corresponding second content representation.

[0055] The second content representation is input into the encoder for encoding processing to obtain the corresponding second intermediate representation;

[0056] The second intermediate representation is input into the variable feature predictor for prediction to obtain the second variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding second variable feature prediction loss.

[0057] The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second true content to calculate the corresponding second Mel spectrum prediction loss.

[0058] In some feasible implementations, the tuning module is specifically used for:

[0059] The acoustic processing unit is parameter-tuned according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until the iteration conditions are met, and the tuned acoustic processing unit is obtained.

[0060] Based on the optimized secondary prosodic representation correction module and the optimized acoustic processing unit, a corresponding acoustic model is constructed.

[0061] In some feasible implementations, the acoustic model further includes at least an acoustic processing unit, and the device further includes:

[0062] The content acquisition module is used to acquire the target phoneme sequence, the target prosodic label, and the target speaker identifier. The target prosodic label includes at least the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label.

[0063] The input module is used to input the target phoneme sequence, the target speaker identifier, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label into the secondary prosodic representation correction module for prosodic label correction, so as to obtain the corrected secondary prosodic label.

[0064] The combination module is used to combine the target phoneme sequence, the target speaker identifier, the predicted primary prosodic label, the corrected secondary prosodic label, and the predicted tertiary prosodic label to obtain a target content representation.

[0065] The synthesis module is used to input the target content representation into the acoustic processing unit for audio synthesis to obtain target audio corresponding to the target factor sequence.

[0066] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0067] The memory is used to store computer programs;

[0068] When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0069] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0070] The embodiments of the present invention have the following advantages:

[0071] In this embodiment of the invention, training data for acoustic model training is obtained. The training data includes at least text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs. The acoustic model includes at least a secondary prosodic representation correction module. First, the text content, audio pairs, acoustic features, and text prosodic features are input into the secondary prosodic representation correction module for prosodic label correction. Based on the correction results, the parameters of the secondary prosodic representation correction module are optimized to obtain the optimized secondary prosodic representation correction module. Then, the model is trained based on the text content, audio pairs, acoustic features, and the optimized secondary prosodic representation correction module. The acoustic model is trained to obtain at least two loss functions. Then, the parameters of the acoustic processing unit are tuned according to the at least two loss functions to obtain an acoustic model that meets the preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker. By constructing the corresponding training data, the same text prosodic features are used in the training process of the acoustic model, that is, the same prosodic distribution is used for model training, which provides the possibility for the correction of the prosodic distribution. At the same time, the model is optimized and a two-level prosodic representation correction module is constructed to support prosodic correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosodic style of different users, while ensuring accurate correspondence with real audio. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the prosodic hierarchy provided in an embodiment of the present invention;

[0073] Figure 2 This is a flowchart of the steps of an acoustic model generation method provided in an embodiment of the present invention;

[0074] Figure 3 This is a schematic diagram of a data sample provided in an embodiment of the present invention;

[0075] Figure 4 This is a schematic diagram of the acoustic model provided in the embodiments of the present invention;

[0076] Figure 5 This is a structural block diagram of an acoustic model generation device provided in an embodiment of the present invention. Detailed Implementation

[0077] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, some technical features involved in the embodiments of the present invention are explained and described below:

[0079] Prosodic structure prediction: In phonetics, based on features such as rhythm, stress, and intonation, the hierarchical structure of prosody can be divided. Chinese phonetics research generally employs a four-tiered prosodic structure: prosodic words, prosodic phrases, intonation phrases, and sentences. For example, referring to… Figure 1 This diagram illustrates the prosodic hierarchy provided in this embodiment of the invention, showing the prosodic hierarchy of a sentence of audio text, simplified using digital tags (#1, #2, #3, #4), namely, "These #1 shoes #2 are not sturdy #3, don't #1 wear them #4." As can be seen from the diagram, the division of prosodic words closely resembles grammatical word segmentation structures. The boundaries of intonation phrases mostly coincide with punctuation marks such as commas, while the sentence hierarchy always coincides with the sentence-ending punctuation. Prosodic structure prediction is the process of predicting the prosodic digital tags of audio text using methods such as rules or neural networks. In practice, prosodic structure prediction is generally performed on single sentences of text, and the sentence tag (#4) appears only at the end of the sentence; therefore, #4 does not need to be predicted. Among the other three levels of prosodic structure prediction, prosodic words (#1) and intonation phrases (#3) typically have high accuracy, with a harmonic average exceeding 95%, while the harmonic average for prosodic phrases (#2) is generally around 75% to 85%. This is because prosodic phrases are highly subjective; different speakers may have different ways of classifying prosodic phrases when pronouncing the same text, and there is often no single correct classification.

[0080] As an example, with the development of large-scale speech synthesis models, scaling laws reveal that the quality and naturalness of synthesized audio improve significantly with increasing training data volume. This prompts us to utilize unprosodic-annotated audio data as much as possible at low cost, but it also introduces new problems. If explicit prosodic structure information is not used at all to train the acoustic model, the correctness of semantic pauses in the output audio cannot be guaranteed, and even intonation phrase (#3) segmentation will be incorrect, greatly affecting comprehension. Even large-scale speech synthesis models trained with a large amount of audio data (such as Vall-E, Cosyvoice, etc.) exhibit frequent semantic pause errors. For acoustic models that use explicit prosodic information for modeling, we consider the use of two main types of audio databases to analyze the problem. The first type, "prosodic-annotated" audio databases, refers to audio databases that correspond to reference text and audio and are annotated with complete prosodic structure labels. The second type, "unprosodic-annotated" audio databases, refers to audio databases that only contain the text corresponding to the audio and have no prosodic structure annotations. The latter type of data is typically automatically labeled using speech recognition. Here, it's assumed that this type of audio database ensures that punctuation marks in the text align with the audio; that is, at the points where there are inter-sentence punctuation marks (such as periods, exclamation marks, question marks, etc.) and commas, the corresponding audio will have a relatively long silent segment or a noticeable pause. When using this second type of data, a separately trained prosodic prediction model is usually used to pre-label the text with prosodic tags before training the acoustic model. However, in the process of building the corresponding acoustic model, there are problems such as the inability to reflect the different prosodic styles of different speakers and the inability to guarantee an accurate correspondence with real audio.

[0081] In this invention, training data for acoustic model training is acquired, including at least text content, corresponding audio pairs, text prosodic features, and acoustic features. The acoustic model includes at least a secondary prosodic representation correction module. First, the text content, audio pairs, acoustic features, and text prosodic features are input into the secondary prosodic representation correction module for prosodic label correction. Based on the correction results, the parameters of the secondary prosodic representation correction module are optimized to obtain the optimized module. Then, the model is trained based on the text content, audio pairs, acoustic features, and the optimized module. The acoustic model is trained to obtain at least two loss functions. Then, the parameters of the acoustic processing unit are tuned according to the at least two loss functions to obtain an acoustic model that meets the preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker. By constructing the corresponding training data, the same text prosodic features are used in the training process of the acoustic model, that is, the same prosodic distribution is used for model training, which provides the possibility for the correction of the prosodic distribution. At the same time, the model is optimized and a two-level prosodic representation correction module is constructed to support prosodic correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosodic style of different users, while ensuring accurate correspondence with real audio.

[0082] Reference Figure 2 The diagram illustrates a flowchart of a method for generating an acoustic model according to an embodiment of the present invention, which may specifically include the following steps:

[0083] Step 201: Obtain training data for training the acoustic model. The training data includes at least text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs. The acoustic model includes at least a two-level prosodic representation correction module and an acoustic processing unit.

[0084] In this embodiment of the invention, training data corresponding to the acoustic model based on the speaker-corrected prosodic label can be constructed first. This is achieved by acquiring data such as text content, corresponding audio pairs, text prosodic features, and acoustic features corresponding to the audio pairs as training data to train the acoustic model. Here, text content refers to the natural language text input into the acoustic model, which can be a sentence, paragraph, or entire document; audio pairs refer to speech audio data corresponding to specific text content, and this invention does not impose any limitations on this.

[0085] In the specific implementation, for each collected text / audio pair, necessary acoustic features (speaker ID, Mel spectrum, fundamental frequency, energy, etc.) and text prosodic features need to be provided. Furthermore, to ensure the consistency of prosodic distribution during training and inference, this embodiment of the invention uniformly uses the prosodic labels predicted by the prosodic model as the model input. Simultaneously, to ensure the correspondence between prosodic labels and audio during training as much as possible, each prosodic level label is processed separately:

[0086] For label #1, the prediction accuracy is considered high, so the prediction result can be used directly. For label #3, it is necessary to ensure that all longer silent segments in the training audio correspond to the predicted #3. This can be achieved by using commas to mark the positions of longer silent segments in the audio, and it is assumed that the prosody prediction model can accurately predict the prosody at the comma as #3. However, for label #2, which is highly subjective and has a low prediction accuracy, this patent introduces a modified #2 to guide model training and inference. Therefore, in the input part, not only the predicted #2 needs to be given, but also a portion of the true #2 labels.

[0087] The specific input content corresponding to the text prosodic features is as follows:

[0088] "Prosodic Marker" phonetic library: phoneme sequence, predicted primary prosodic label ID, predicted secondary prosodic label ID, predicted tertiary prosodic label ID, and actual secondary prosodic label ID.

[0089] "Unprosodic labeling" phoneme library: phoneme sequence, predicted primary prosodic label ID, predicted secondary prosodic label ID, predicted tertiary prosodic label ID (optional: actual secondary prosodic label ID).

[0090] Based on the differences in text prosodic features, the training data can be divided into two groups. The first group consists of labeled data containing real secondary prosodic label IDs, and the second group consists of unlabeled data that does not contain real secondary prosodic label IDs. The acoustic model is trained based on the labeled and unlabeled data. This ensures that for "prosodic-labeled" audio databases, the same prosodic distribution is used as input during acoustic model training and inference. For "unlabeled" audio databases, this reduces the gap between the input prosodic labels and the original audio prosodics when participating in acoustic model training, thereby improving prediction accuracy.

[0091] In addition, for audio libraries with "non-prosodic annotation", a small portion of the audio text pairs can be manually annotated with real #2 tags, and semi-supervised training can be carried out in subsequent experiments.

[0092] In one example, refer to Figure 3The diagram illustrates a data example provided in an embodiment of the present invention, wherein sil in the phoneme sequence represents the start and end silence segment, sp3 represents the long silence segment corresponding to #3 in the sentence, and ^ represents the zero initial phoneme; 0 in the prosodic label ID represents non-boundary and 1 represents boundary.

[0093] It should be noted that a phoneme sequence refers to a set of phonemes used in a language to represent the pronunciation of a specific word, phrase, or sentence. A phoneme is the smallest unit in phonetics, capable of distinguishing meaning; that is, two different phonemes may correspond to two different word meanings in the same language.

[0094] For example, in English, the word "cat" can be broken down into three phonemes: / k / , / æ / , and / t / . Similarly, the word "bat" can be broken down into three phonemes: / b / , / æ / , and / t / . Although both words have a / æ / and a / t / in their phoneme sequences, their meanings differ because the first phoneme is different ( / k / and / b / ).

[0095] After constructing the corresponding training data, an acoustic model to be trained can be obtained. The acoustic model can be used to synthesize the target audio corresponding to the speaker. Optionally, the acoustic model can include an input layer, a secondary prosodic representation correction module, and an acoustic processing unit. The input layer can be used to input the corresponding training data, the secondary prosodic representation correction module can be used to correct the corresponding prosodic labels to alleviate the inconsistency between text prosodic features and audio when training with unlabeled data, and the acoustic processing unit can be used to encode, predict, and decode the mapped vector features to synthesize the corresponding target audio.

[0096] Step 202: Input the text content, the audio pair data, the acoustic features, and the text prosodic features into the secondary prosodic representation correction module to correct the prosodic labels. Based on the correction results, perform parameter optimization on the secondary prosodic representation correction module to obtain the optimized secondary prosodic representation correction module.

[0097] During the training of the acoustic model, the parameters of the secondary prosodic representation correction module can be optimized first. Specifically, the text content, audio pairs, acoustic features, and text prosodic features can be input into the secondary prosodic representation correction module to correct the prosodic labels. Based on the correction results, the parameters of the secondary prosodic representation correction module are optimized to obtain the optimized secondary prosodic representation correction module. The secondary prosodic representation correction module can correct the secondary prosodic labels in the prediction process, thereby alleviating the problem of inconsistency between text prosodic features and audio, and ensuring accurate correspondence with real audio.

[0098] It should be noted that since the training data includes both labeled and unlabeled data, the labeled data contains true secondary prosodic labels. These true secondary prosodic labels can be compared with the secondary prosodic labels predicted by the secondary prosodic representation correction module during training to calculate the corresponding prosodic label prediction loss. Then, the parameters of the secondary prosodic representation correction module are tuned based on the prosodic label prediction loss to optimize the model parameters corresponding to the secondary prosodic representation correction module. However, for unlabeled data, since it does not contain true secondary prosodic labels, the prosodic label prediction loss obtained when training the secondary prosodic representation correction module using unlabeled data is 0. That is, during training, the model can be tuned based solely on labeled data for the secondary prosodic representation correction module.

[0099] In a specific implementation, the text prosodic features include at least a first type of prosodic label. The first type of prosodic label includes at least a predicted first-level prosodic label, a predicted second-level prosodic label, a predicted third-level prosodic label, and a true second-level prosodic label. Then, the text content, audio pair data, acoustic features, predicted first-level prosodic labels, predicted second-level prosodic labels, and predicted third-level prosodic labels are first input into the second-level prosodic representation correction module for prediction to obtain the corresponding prediction results. Then, the prediction results are compared with the true second-level prosodic labels to determine the prosodic label prediction loss for the second-level prosodic representation correction module. Then, the prosodic label prediction loss is used to optimize the parameters of the second-level prosodic representation correction module until the prosodic label prediction loss meets the preset conditions, and the optimized second-level prosodic representation correction module is obtained.

[0100] Optionally, if the acoustic features include at least the speaker identifier, the feature input process can also be as follows: obtain the phoneme features corresponding to the audio pair data, then input the text content, speaker identifier, phoneme features, predicted first-level prosodic label, predicted second-level prosodic label, and predicted third-level prosodic label into the second-level prosodic representation correction module for prediction, obtain the corresponding prediction results, and then perform the model tuning process as described in the above embodiment, thereby alleviating the problem of inconsistency between text prosodic features and audio, and ensuring accurate correspondence with real audio.

[0101] It should be noted that, in this embodiment of the invention, an acoustic model with a FastPitch-like structure can be used for modeling, with modifications made to its encoder. Specifically, the encoder mainly adds prosodic features and introduces a second-level prosodic label corrector module (i.e., a second-level prosodic representation correction module) before adding the phoneme representation to the prosodic representations at each level. This module first corrects the predicted second-level prosodic phrase IDs and then uses the corrected second-level prosodic phrase IDs to obtain the final second-level prosodic representation. The key component in the acoustic model is the second-level prosodic label corrector (i.e., the second-level prosodic representation correction module). The input conditions include text content, speaker information, and predicted first, second, and third-level prosodic label IDs. The output is the corrected second-level prosodic label ID for the speaker on the corresponding text. The model structure is the same as the duration predictor of FastPitch, consisting of two one-dimensional convolutional layers.

[0102] In one example, refer to Figure 4 The diagram illustrates the structure of the acoustic model provided in this embodiment of the invention. For the model optimization process of the secondary prosodic representation correction module (i.e., the secondary prosodic representation correction module), feature extraction can first be performed on the speaker ID to obtain the corresponding speaker representation (e.g., extracting the acoustic features corresponding to the speaker from the audio), and feature extraction can be performed on the phoneme ID to obtain the corresponding phoneme representation (e.g., extracting the corresponding phoneme features from the text content). Then, the predicted primary prosodic label ID, the predicted secondary prosodic label ID, and the predicted tertiary prosodic label ID are all input into the secondary prosodic representation correction module for prediction, obtaining the predicted... After obtaining the predicted secondary prosodic label ID, the predicted secondary prosodic label ID is compared with the actual secondary prosodic label ID. The prediction loss of the secondary prosodic label is calculated, and then the model of the secondary prosodic representation correction module is optimized in reverse based on the prediction loss of the secondary prosodic label. The iteration of the secondary prosodic representation correction module is stopped when the prediction loss of the secondary prosodic label is less than or equal to a preset threshold. The optimized secondary prosodic representation correction module is obtained. Thus, by introducing secondary prosodic label correction, prosodic correction based on the characteristics of the speaker can be supported, which helps to alleviate the problem of inconsistency between text prosodic features and audio, and ensures accurate correspondence with real audio.

[0103] Step 203: Train the model based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions;

[0104] After completing the model tuning of the secondary prosodic representation correction module, the model can be further trained based on the training data and the tuned secondary prosodic representation correction module to obtain at least two corresponding loss functions. Then, the acoustic processing unit in the acoustic model can be tuned based on at least two loss functions to obtain the corresponding acoustic model.

[0105] In this embodiment of the invention, the acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor. The encoder can encode the input features to obtain a high-dimensional representation, and then input the high-dimensional representation to the variable feature predictor to predict the variable features of the audio, such as pitch and energy, and input the variable features to the decoder for decoding, so as to decode the content representation, prosodic representation and variable features into the corresponding Mel spectrum, thereby synthesizing the corresponding audio.

[0106] For labeled data, the text content, audio pairs, acoustic features, predicted primary prosodic labels, predicted secondary prosodic labels, and predicted tertiary prosodic labels can be input into the optimized secondary prosodic representation correction module to obtain the corrected secondary prosodic labels. Then, the text content, audio pairs, acoustic features, predicted primary prosodic labels, corrected secondary prosodic labels, and predicted tertiary prosodic labels are combined to obtain the corresponding first content representation. The first content representation is then input into the encoder for encoding to obtain the corresponding first intermediate representation. Next, the first intermediate representation is input into the variable feature predictor for prediction to obtain the first variable feature related to speech. The variable feature is then compared with the preset first real content to calculate the corresponding first variable feature prediction loss. Finally, the variable feature is input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second real content to calculate the corresponding first Mel spectrum prediction loss.

[0107] For unlabeled data, text prosodic features include at least a second type of prosodic label, which includes at least predicted primary prosodic labels, predicted secondary prosodic labels, and predicted tertiary prosodic labels. The corresponding training process can first combine text content, audio pairs, acoustic features, predicted primary prosodic labels, predicted secondary prosodic labels, and predicted tertiary prosodic labels to obtain the corresponding second content representation. The second content representation is then input into the encoder for encoding to obtain the corresponding second intermediate representation. Next, the second intermediate representation is input into the variable feature predictor for prediction to obtain the second variable features related to speech. The variable features are then compared with the preset first real content to calculate the corresponding second variable feature prediction loss. Finally, the variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second real content to calculate the corresponding second Mel spectrum prediction loss.

[0108] It should be noted that when inputting data into the model without manually labeled secondary prosodic tags, updating parameters solely through Mel-spectrum prediction loss and variable feature prediction loss indirectly affects the secondary prosodic tag corrector. That is, the corrected secondary prosodic tags should help reduce the Mel-spectrum prediction loss and aid in predicting the variable features of the current audio. By providing the possibility of modifying secondary prosodic tags, the inconsistency between text prosodic features and audio during training with unlabeled data is alleviated.

[0109] Step 204: The acoustic processing unit is parameter-tuned according to at least two of the loss functions to obtain an acoustic model that meets preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker.

[0110] After obtaining loss functions such as variable feature prediction loss and Mel spectrum prediction loss through the above process, the acoustic processing unit can be comprehensively optimized based on at least two of the obtained loss functions. Parameter optimization is stopped when all loss functions meet the stopping iteration condition, resulting in an optimized acoustic processing unit. Based on the optimized secondary prosodic representation correction module and the optimized acoustic processing unit, a corresponding acoustic model is constructed. By constructing corresponding training data, the same text prosodic features (i.e., the same prosodic distribution) are used during the training of the acoustic model, providing the possibility for prosodic distribution correction. Simultaneously, the model is optimized, and a secondary prosodic representation correction module is constructed, supporting prosodic correction based on user characteristics. This ensures that the trained acoustic model accurately reflects the prosodic style of different users while guaranteeing precise correspondence with real audio.

[0111] In the specific implementation, based on the obtained loss function, the acoustic processing unit can be parameter-tuned according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until the iteration conditions are met, thus obtaining the optimized acoustic processing unit. Then, based on the optimized secondary prosodic representation correction module and the optimized acoustic processing unit, the corresponding acoustic model is constructed. By constructing the corresponding training data, the same text prosodic features are used in the training process of the acoustic model, that is, the same prosodic distribution is used for model training, which provides the possibility for the correction of the prosodic distribution. At the same time, the model is optimized and a secondary prosodic representation correction module is constructed to support prosodic correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosodic style of different users, while ensuring accurate correspondence with real audio.

[0112] It should be noted that for variable feature prediction loss, Mel spectrum prediction loss, etc., corresponding thresholds can be set respectively. When the loss function is less than or equal to the preset threshold, the model iteration can be stopped and the model tuning process can be completed. For example, a first threshold can be set for variable feature prediction loss and a second threshold can be set for Mel spectrum prediction loss. Whether the model is trained with labeled data or unlabeled data, when the variable feature prediction loss is less than or equal to the first threshold and the Mel spectrum prediction loss is less than or equal to the second threshold, the parameter tuning of the acoustic processing unit can be stopped and the model tuning process can be completed. In order to combine the tuned secondary prosodic representation correction module, the corresponding acoustic model can be constructed.

[0113] Based on the above process, after training the corresponding acoustic model, audio synthesis can be performed using the acoustic model. Specifically, this can be achieved by acquiring the target phoneme sequence, target prosodic tags, and target speaker identifiers. The target prosodic tags include at least predicted primary, secondary, and tertiary prosodic tags. Then, the target phoneme sequence, target speaker identifier, and the predicted primary, secondary, and tertiary prosodic tags are input into the secondary prosodic representation correction module for prosodic tag correction, obtaining the corrected secondary prosodic tags. Finally, the target phoneme sequence, target speaker identifier, and the predicted primary and tertiary prosodic tags are input into the secondary prosodic representation correction module for prosodic tag correction. The corrected secondary prosodic labels and the predicted tertiary prosodic labels are combined to obtain the target content representation. The target content representation is then input into the acoustic processing unit for audio synthesis to obtain the target audio corresponding to the target factor sequence. After being corrected by the secondary prosodic label corrector, a secondary prosodic pause that conforms to the speaker's style is obtained. Finally, the model synthesizes the Mel spectrum that conforms to the corresponding prosodic, and the vocoder outputs the corresponding audio. At the same time, for the same input text, it can obtain relatively accurate primary and tertiary prosodic labels based on the general prosodic model, and can also fine-tune the secondary prosodic labels according to the prosodic characteristics of different speakers to achieve intra-sentence pauses, making the synthesis result more in line with the speaker's style.

[0114] It should be noted that the embodiments of the present invention include, but are not limited to, the examples described above. It is understood that those skilled in the art can make further settings according to actual needs under the guidance of the ideas in the embodiments of the present invention, and the present invention does not limit such settings.

[0115] In this embodiment of the invention, training data for acoustic model training is obtained. The training data includes at least text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs. The acoustic model includes at least a secondary prosodic representation correction module. First, the text content, audio pairs, acoustic features, and text prosodic features are input into the secondary prosodic representation correction module for prosodic label correction. Based on the correction results, the parameters of the secondary prosodic representation correction module are optimized to obtain the optimized secondary prosodic representation correction module. Then, the model is trained based on the text content, audio pairs, acoustic features, and the optimized secondary prosodic representation correction module. The acoustic model is trained to obtain at least two loss functions. Then, the parameters of the acoustic processing unit are tuned according to the at least two loss functions to obtain an acoustic model that meets the preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker. By constructing the corresponding training data, the same text prosodic features are used in the training process of the acoustic model, that is, the same prosodic distribution is used for model training, which provides the possibility for the correction of the prosodic distribution. At the same time, the model is optimized and a two-level prosodic representation correction module is constructed to support prosodic correction according to user characteristics, so that the trained acoustic model can accurately reflect the prosodic style of different users, while ensuring accurate correspondence with real audio.

[0116] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are provided for illustrative purposes:

[0117] Reference Figure 4 The acoustic model shown can be constructed through the following process:

[0118] S01. Construct training data for an acoustic model based on speaker-modified prosodic labels.

[0119] For each text and audio pair of data collected, it is necessary to provide the necessary acoustic features (speaker ID, Mel spectrum, fundamental frequency, energy, etc.) and text prosodic features.

[0120] To ensure consistency in prosodic distribution during training and inference, this patent uniformly uses the prosodic labels predicted by the prosodic model as the model input. To maintain the correspondence between prosodic labels and audio during training as much as possible, each prosodic level label is processed separately:

[0121] For label #1, the prediction accuracy is considered to be high, so the prediction result can be used directly.

[0122] For the #3 label, it is necessary to ensure that all longer silent segments in the training audio correspond to the predicted #3. This can be achieved by using commas to mark the positions of longer silent segments in the audio, and it is assumed here that the prosody prediction model can accurately predict the prosody at the comma as #3.

[0123] However, for the #2 label, which is highly subjective and has low prediction accuracy, this patent introduces a modified #2 to guide model training and inference. Therefore, the input part needs to provide not only the predicted #2 but also a portion of the actual #2 label. The specific text prosody input content is as follows:

[0124] "Prosodic Marker" phonetic library: phoneme sequence, predicted primary prosodic label ID, predicted secondary prosodic label ID, predicted tertiary prosodic label ID, and actual secondary prosodic label ID.

[0125] "Unprosodic labeling" phoneme library: phoneme sequence, predicted primary prosodic label ID, predicted secondary prosodic label ID, predicted tertiary prosodic label ID (optional: actual secondary prosodic label ID).

[0126] Among them, the "non-prosodic annotation" audio library can select a small portion of audio text pairs for manual annotation with real #2 tags, and conduct semi-supervised training in subsequent experiments.

[0127] In this context, sil in the phoneme sequence represents the start and end of a silent segment, sp3 represents the long silent segment corresponding to #3 in the sentence, and ^ represents a zero initial phoneme; 0 in the prosodic label ID represents a non-boundary segment, and 1 represents a boundary segment.

[0128] S02. Construct an acoustic model based on the speaker's modified prosodic label.

[0129] This patent uses a FastPitch-like acoustic model for modeling and modifies its encoder. The main modification to the encoder is the addition of prosodic representation, and the introduction of a second-level prosodic label corrector module before adding the phoneme representation to the prosodic representations at each level. This module first corrects the predicted second-level prosodic phrase IDs, and then uses the corrected second-level prosodic phrase IDs to obtain the final second-level prosodic representation.

[0130] The key component is the secondary prosodic label corrector. The input conditions include text content, speaker information, and predicted primary, secondary, and tertiary prosodic label IDs. The output is the secondary prosodic label ID corrected by the speaker on the corresponding text. The model structure is the same as FastPitch's duration predictor, which consists of two one-dimensional convolutional layers.

[0131] S03. Add a semi-supervised second-level prosodic label prediction loss and train the model parameters.

[0132] Based on the original FastPitch model's training losses (Mel spectrum prediction loss and variable feature prediction loss), a cross-entropy loss from second-level prosodic label prediction is added to jointly guide the training of the constructed acoustic model. The second-level prosodic label prediction loss is primarily used to directly guide the learning of the second-level prosodic label modifier.

[0133] For most data in the "unannotated" audio database that lacks manually labeled secondary prosodic tags, the corresponding secondary prosodic tag prediction loss is 0, while the loss for other data is greater than 0. When inputting data without manually labeled secondary prosodic tags into the model, updating parameters solely through Mel-spectrum prediction loss and variable feature prediction loss indirectly affects the secondary prosodic tag corrector. That is, the corrected secondary prosodic tags should help reduce Mel-spectrum prediction loss and aid in predicting the variable features of the current audio.

[0134] By providing the possibility of modifying secondary prosodic labels, the inconsistency between text prosodic features and audio during training with unlabeled data is alleviated.

[0135] S04. Inference supports the synthesis of different prosody by different speakers.

[0136] Using the acoustic model trained above, the input phoneme sequence ID, predicted first, second, and third-level prosodic label IDs, and speaker ID are processed by a second-level prosodic label corrector to obtain second-level prosodic pauses that match the speaker's style. Finally, the model synthesizes a Mel spectrum that conforms to the corresponding prosodic pattern, and the vocoder outputs the corresponding audio. Thus, for the same input text, it can obtain relatively accurate #1 and #3 based on a general prosodic model, and can also fine-tune the #2 intra-sentence pauses according to the prosodic characteristics of different speakers, making the synthesized result more closely match the speaker's style.

[0137] In the aforementioned process, a unique prosodic feature was constructed for the acoustic model training data. This feature utilized both hierarchical prosodic prediction labels identical to those used during inference and optional manually calibrated secondary prosodic labels. This enables real-time correction of the prosodic distribution based on content, speaker, and other information. Furthermore, a secondary prosodic label corrector was introduced into the FastPitch-like acoustic model structure to support prosodic correction based on speaker characteristics. Additionally, semi-supervised learning supporting secondary prosodic label modification indirectly improves the accuracy of the modified prosodic labels compared to the original audio prosodic by reducing alignment and Mel loss.

[0138] Based on the above process, the technical solutions involved in the embodiments of the present invention can achieve at least the following beneficial effects:

[0139] For "prosodic-labeled" audio databases, the same prosodic distribution is used as input during acoustic model training and inference. For "unlabeled" audio databases, the gap between the input prosodic labels and the original audio prosodics is reduced when they participate in acoustic model training. During acoustic model inference, unique and stable prosodic effects are synthesized based on the characteristics of the speaker.

[0140] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0141] Reference Figure 5 The diagram shows a structural block diagram of an acoustic model generation device provided in an embodiment of the present invention, which may specifically include the following modules:

[0142] The data acquisition module 501 is used to acquire training data for training the acoustic model. The training data includes at least text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs. The acoustic model includes at least a two-level prosodic representation correction module and an acoustic processing unit.

[0143] The correction module 502 is used to input the text content, the audio pair data, the acoustic features and the text prosodic features into the secondary prosodic representation correction module to correct the prosodic labels, and to optimize the parameters of the secondary prosodic representation correction module based on the correction results to obtain the optimized secondary prosodic representation correction module.

[0144] Training module 503 is used to train the model based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions;

[0145] The tuning module 504 is used to tune the parameters of the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker.

[0146] In some feasible implementations, the text prosodic features include at least a first type of prosodic label, which includes at least a predicted primary prosodic label, a predicted secondary prosodic label, a predicted tertiary prosodic label, and a true secondary prosodic label. The correction module 502 is specifically used for:

[0147] The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prediction to obtain the corresponding prediction results.

[0148] The prediction results are compared with the actual secondary prosodic labels to determine the prosodic label prediction loss for the secondary prosodic representation correction module.

[0149] The prosodic label prediction loss is used to fine-tune the parameters of the secondary prosodic representation correction module until the prosodic label prediction loss meets the preset conditions, thus obtaining the fine-tuned secondary prosodic representation correction module.

[0150] In some feasible implementations, the acoustic features include at least a speaker identifier, and the correction module 502 is specifically used for:

[0151] Obtain the phoneme features corresponding to the audio pair data;

[0152] The text content, the speaker identifier, the phoneme features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prediction to obtain the corresponding prediction results.

[0153] In some feasible implementations, the acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor. The training module 503 is specifically used for:

[0154] The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the optimized secondary prosodic representation correction module to obtain the corrected secondary prosodic label.

[0155] The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the corrected secondary prosodic label, and the predicted tertiary prosodic label are combined to obtain the corresponding first content representation.

[0156] The first content representation is input into the encoder for encoding processing to obtain the corresponding first intermediate representation;

[0157] The first intermediate representation is input into the variable feature predictor for prediction to obtain the first variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding first variable feature prediction loss.

[0158] The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second real content to calculate the corresponding first Mel spectrum prediction loss.

[0159] In some feasible implementations, the text prosodic features include at least a second type of prosodic label, which includes at least a predicted first-level prosodic label, a predicted second-level prosodic label, and a predicted third-level prosodic label. The training module 503 is specifically used for:

[0160] The corresponding second content representation is obtained by combining the text content, the audio pair data, the acoustic features, the predicted first-level prosodic label, the predicted second-level prosodic label, and the predicted third-level prosodic label.

[0161] The second content representation is input into the encoder for encoding processing to obtain the corresponding second intermediate representation;

[0162] The second intermediate representation is input into the variable feature predictor for prediction to obtain the second variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding second variable feature prediction loss.

[0163] The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second true content to calculate the corresponding second Mel spectrum prediction loss.

[0164] In some feasible implementations, the tuning module 504 is specifically used for:

[0165] The acoustic processing unit is parameter-tuned according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until the iteration conditions are met, and the tuned acoustic processing unit is obtained.

[0166] Based on the optimized secondary prosodic representation correction module and the optimized acoustic processing unit, a corresponding acoustic model is constructed.

[0167] In some feasible implementations, the acoustic model further includes at least an acoustic processing unit, and the device further includes:

[0168] The content acquisition module is used to acquire the target phoneme sequence, the target prosodic label, and the target speaker identifier. The target prosodic label includes at least the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label.

[0169] The input module is used to input the target phoneme sequence, the target speaker identifier, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label into the secondary prosodic representation correction module for prosodic label correction, so as to obtain the corrected secondary prosodic label.

[0170] The combination module is used to combine the target phoneme sequence, the target speaker identifier, the predicted primary prosodic label, the corrected secondary prosodic label, and the predicted tertiary prosodic label to obtain a target content representation.

[0171] The synthesis module is used to input the target content representation into the acoustic processing unit for audio synthesis to obtain target audio corresponding to the target factor sequence.

[0172] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0173] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described acoustic model generation method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0174] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described acoustic model generation method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0175] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0176] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program code.

[0177] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0181] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0182] The above provides a detailed description of the method and apparatus for generating an acoustic model provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating an acoustic model, characterized in that, include: Acquire training data for training an acoustic model. The training data includes at least text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs. The acoustic model includes at least a two-level prosodic representation correction module and an acoustic processing unit. The acoustic features include at least a speaker identifier. The text content, the audio pair data, the speaker identifier, and the text prosodic features are input into the secondary prosodic representation correction module to correct the prosodic labels. Based on the correction results, the parameters of the secondary prosodic representation correction module are optimized to obtain the optimized secondary prosodic representation correction module. The model is trained based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions. The acoustic processing unit is parameter-tuned according to at least two of the loss functions to obtain an acoustic model that meets preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker. The acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor. The model is trained based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions, including: The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the optimized secondary prosodic representation correction module to obtain the corrected secondary prosodic label. The primary prosodic label is used to represent prosodic words, the secondary prosodic label is used to represent prosodic phrases, and the tertiary prosodic label is used to represent intonation phrases. The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the corrected secondary prosodic label, and the predicted tertiary prosodic label are combined to obtain the corresponding first content representation. The first content representation is input into the encoder for encoding processing to obtain the corresponding first intermediate representation; The first intermediate representation is input into the variable feature predictor for prediction to obtain the first variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding first variable feature prediction loss. The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second real content to calculate the corresponding first Mel spectrum prediction loss.

2. The method according to claim 1, characterized in that, The text prosodic features include at least a first type of prosodic label, which includes at least a predicted primary prosodic label, a predicted secondary prosodic label, a predicted tertiary prosodic label, and a true secondary prosodic label. The process involves inputting the text content, the audio pair data, the speaker identifier, and the text prosodic features into the secondary prosodic representation correction module to correct the prosodic labels. Based on the correction results, the parameters of the secondary prosodic representation correction module are optimized to obtain the optimized secondary prosodic representation correction module, including: The text content, the audio pair data, the speaker identifier, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prediction to obtain the corresponding prediction results. The prediction results are compared with the actual secondary prosodic labels to determine the prosodic label prediction loss for the secondary prosodic representation correction module. The prosodic label prediction loss is used to fine-tune the parameters of the secondary prosodic representation correction module until the prosodic label prediction loss meets the preset conditions, thus obtaining the fine-tuned secondary prosodic representation correction module.

3. The method according to claim 2, characterized in that, The process of inputting the text content, the audio pair data, the speaker identifier, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label into the secondary prosodic representation correction module for prediction, and obtaining the corresponding prediction result, includes: Obtain the phoneme features corresponding to the audio pair data; The text content, the speaker identifier, the phoneme features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prediction to obtain the corresponding prediction results.

4. The method according to claim 2 or 3, characterized in that, The text prosodic features include at least a second type of prosodic label, which includes at least a predicted primary prosodic label, a predicted secondary prosodic label, and a predicted tertiary prosodic label. The model is trained based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions, including: The text content, the audio pair data, the acoustic features, the predicted first-level prosodic labels, the predicted second-level prosodic labels, and the predicted third-level prosodic labels are combined to obtain the corresponding second content representation. The second content representation is input into the encoder for encoding processing to obtain the corresponding second intermediate representation; The second intermediate representation is input into the variable feature predictor for prediction to obtain the second variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding second variable feature prediction loss. The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second true content to calculate the corresponding second Mel spectrum prediction loss.

5. The method according to claim 4, characterized in that, The step of optimizing the parameters of the acoustic processing unit according to at least two of the aforementioned loss functions to obtain an acoustic model that meets preset conditions includes: The acoustic processing unit is parameter-tuned according to the first variable feature prediction loss, the second variable feature prediction loss, the first Mel spectrum prediction loss, and the second Mel spectrum prediction loss until the iteration conditions are met, and the tuned acoustic processing unit is obtained. Based on the optimized secondary prosodic representation correction module and the optimized acoustic processing unit, a corresponding acoustic model is constructed.

6. The method according to claim 1, characterized in that, The acoustic model further includes at least an acoustic processing unit, and the method further includes: Obtain the target phoneme sequence, target prosodic label, and target speaker identifier, wherein the target prosodic label includes at least the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label; The target phoneme sequence, the target speaker identifier, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the secondary prosodic representation correction module for prosodic label correction to obtain the corrected secondary prosodic label. The target phoneme sequence, the target speaker identifier, the predicted primary prosodic label, the corrected secondary prosodic label, and the predicted tertiary prosodic label are combined to obtain the target content representation. The target content representation is input into the acoustic processing unit for audio synthesis to obtain the target audio corresponding to the target phoneme sequence.

7. An apparatus for generating an acoustic model, characterized in that, include: The data acquisition module is used to acquire training data for training the acoustic model. The training data includes at least text content, audio pairs corresponding to the text content, text prosodic features corresponding to the text content, and acoustic features corresponding to the audio pairs. The acoustic model includes at least a two-level prosodic representation correction module and an acoustic processing unit. The acoustic features include at least a speaker identifier. The correction module is used to input the text content, the audio pair data, the speaker identifier and the text prosodic features into the secondary prosodic representation correction module to correct the prosodic label, and to optimize the parameters of the secondary prosodic representation correction module based on the correction result to obtain the optimized secondary prosodic representation correction module. The training module is used to train the model based on the text content, the audio pair data, the acoustic features, and the optimized secondary prosodic representation correction module to obtain at least two loss functions. The tuning module is used to tune the parameters of the acoustic processing unit according to at least two of the loss functions to obtain an acoustic model that meets preset conditions. The acoustic model is used to synthesize the target audio corresponding to the speaker. The acoustic processing unit includes at least an encoder, a variable feature predictor connected to the encoder, and a decoder connected to the variable feature predictor. The training module is specifically used for: The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the predicted secondary prosodic label, and the predicted tertiary prosodic label are input into the optimized secondary prosodic representation correction module to obtain the corrected secondary prosodic label. The primary prosodic label is used to represent prosodic words, the secondary prosodic label is used to represent prosodic phrases, and the tertiary prosodic label is used to represent intonation phrases. The text content, the audio pair data, the acoustic features, the predicted primary prosodic label, the corrected secondary prosodic label, and the predicted tertiary prosodic label are combined to obtain the corresponding first content representation. The first content representation is input into the encoder for encoding processing to obtain the corresponding first intermediate representation; The first intermediate representation is input into the variable feature predictor for prediction to obtain the first variable feature related to speech, and the variable feature is compared with the preset first real content to calculate the corresponding first variable feature prediction loss. The variable features are input into the decoder for decoding to obtain the corresponding predicted Mel spectrum. The predicted Mel spectrum is then compared with the preset second real content to calculate the corresponding first Mel spectrum prediction loss.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-6.

Images