Speech synthesis model training method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a speech synthesis model training method and device, electronic equipment and storage medium, comprising: obtaining unannotated paragraph speech, performing speech segmentation processing on the paragraph speech through a first model to obtain multiple single-sentence speeches; obtaining single-sentence texts corresponding to each single-sentence speech, and performing grapheme-to-phoneme conversion processing on the single-sentence texts through a second model to obtain single-sentence phoneme sequences corresponding to each single-sentence speech; taking the multiple single-sentence speeches and the single-sentence phoneme sequences corresponding to each single-sentence speech as standard length training data, training a to-be-trained speech synthesis model based on the standard length training data to obtain a trained speech synthesis model. The method realizes the construction of training data based on unannotated paragraph speech, and can realize the automatic acquisition of single-sentence phoneme sequences and single-sentence speeches in the training data, thereby improving the training efficiency of the speech synthesis model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a speech synthesis model training method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Speech synthesis is a technology that converts received text sequences into natural and realistic speech waveforms. Currently, the mainstream speech synthesis methods include waveform concatenation-based synthesis, statistical acoustic modeling-based synthesis, and end-to-end synthesis. Among these, end-to-end speech synthesis methods have become a hot research topic due to their relatively simple structure, lower requirement for expert knowledge, and language independence, and have achieved successful applications in many fields.

[0003] However, current end-to-end speech synthesis methods require recording speech in a professional environment (such as a recording studio) to build the speech database used for training the speech synthesis model. Furthermore, the recorded speech must maintain consistent speech rate, intonation, emotion, and energy. A professional annotation team is also needed to annotate the recorded speech with text, pinyin, and prosody. Therefore, current speech synthesis models rely heavily on a large amount of annotated speech data for training, resulting in low data preparation efficiency and consequently low training efficiency. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, storage medium, and program product for training a speech synthesis model, to at least solve the problem in related technologies that rely on a large amount of labeled speech data, resulting in low efficiency in training data preparation and consequently low training efficiency of the speech synthesis model. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a method for training a speech synthesis model is provided, comprising:

[0006] Unlabeled speech segments are obtained, and the speech segments are segmented using a first model to obtain multiple single-sentence speech segments.

[0007] The single-sentence text corresponding to each of the multiple single-sentence speech is obtained, and the single-sentence text is processed by the second model to obtain the single-sentence phoneme sequence corresponding to each single-sentence speech.

[0008] The multiple single-sentence speech and the corresponding single-sentence phoneme sequences are used as standard-length training data. The speech synthesis model to be trained is trained based on the standard-length training data to obtain the trained speech synthesis model.

[0009] In one exemplary embodiment, the method further includes:

[0010] At least two single-sentence speech samples from the plurality of single-sentence speech samples are spliced ​​together to obtain corresponding spliced ​​speech samples, and the single-sentence phoneme sequences corresponding to the at least two single-sentence speech samples are spliced ​​together to obtain corresponding spliced ​​phoneme sequences, and corresponding long sentence training data is obtained based on the spliced ​​speech samples and the spliced ​​phoneme sequences;

[0011] At least one of the multiple single-sentence speech segments is segmented to obtain corresponding segmented speech, and the single-sentence phoneme sequence corresponding to the at least one single-sentence speech is segmented to obtain corresponding segmented phoneme sequence, and corresponding short sentence training data is obtained based on the segmented speech and the segmented phoneme sequence.

[0012] The process of training the speech synthesis model to be trained based on the standard-length training data to obtain the trained speech synthesis model includes:

[0013] The speech synthesis model to be trained is trained based on the standard length training data to obtain a pre-trained speech synthesis model.

[0014] Based on the standard-length training data, the long-sentence training data, and the short-sentence training data, the pre-trained speech synthesis model is trained to obtain the trained speech synthesis model.

[0015] In an exemplary embodiment, obtaining the corresponding long sentence training data based on the spliced ​​speech and the spliced ​​phoneme sequence includes:

[0016] The spliced ​​speech is subjected to silence completion processing and / or pruning processing to obtain the processed spliced ​​speech corresponding to the spliced ​​speech; wherein, silence refers to the part of the speech that does not contain the target sound, silence completion processing means adding silence to the speech, and silence pruning processing means discarding part of the silence in the speech.

[0017] The processed and spliced ​​speech and the spliced ​​phoneme sequence are used as training data for the long sentence.

[0018] In an exemplary embodiment, the step of performing silence completion processing and / or trimming processing on the spliced ​​speech to obtain the processed spliced ​​speech includes:

[0019] The initial and final silence durations of the spliced ​​speech are supplemented and / or trimmed to make them the same.

[0020] Furthermore, the silence duration between two adjacent single-sentence speech units that make up the spliced ​​speech is supplemented or trimmed so that the silence duration between the two adjacent single-sentence speech units is the same as the silence duration between two phonemes in each single-sentence speech unit.

[0021] In an exemplary embodiment, the step of segmenting at least one of the plurality of single-sentence speech to obtain corresponding segmented speech, and segmenting the single-sentence phoneme sequence corresponding to the at least one single-sentence speech to obtain a corresponding segmented phoneme sequence, includes:

[0022] The single-sentence phoneme sequences corresponding to each of the multiple single-sentence speech are input into the pre-trained speech synthesis model to obtain the alignment information of the single-sentence phoneme sequences; the alignment information represents the first and last silence durations of the single-sentence phoneme sequences and the speech length corresponding to each phoneme;

[0023] Based on the alignment information, at least one of the multiple single-sentence speech segments is segmented to obtain the corresponding segmented speech; and based on the alignment information, the single-sentence phoneme sequence corresponding to the at least one single-sentence speech segment is segmented to obtain the corresponding segmented phoneme sequence.

[0024] In an exemplary embodiment, obtaining the corresponding short sentence training data based on the segmented speech and the segmented phoneme sequence includes:

[0025] The segmented speech is subjected to silence completion processing and / or pruning processing to obtain the processed segmented speech corresponding to the segmented speech; wherein, silence refers to the part of the speech that does not contain the target sound, silence completion processing means adding silence to the speech, and silence pruning processing means discarding part of the silence in the speech.

[0026] The processed segmented speech and the segmented phoneme sequence are used as training data for the short sentence.

[0027] In an exemplary embodiment, the step of performing silence completion processing and / or cropping processing on the segmented speech to obtain the processed segmented speech includes:

[0028] The silence duration at the beginning and end of the segmented speech is completed and / or trimmed to make the silence duration at the beginning and end of the segmented speech the same.

[0029] In an exemplary embodiment, training the pre-trained speech synthesis model based on the standard-length training data, the long sentence training data, and the short sentence training data to obtain the trained speech synthesis model includes:

[0030] The pre-trained speech synthesis model is trained using the short sentence training data to obtain a retrained speech synthesis model.

[0031] The retrained speech synthesis model is trained using the standard-length training data, the short-sentence training data, and the long-sentence training data until a fully trained speech synthesis model is obtained.

[0032] In one exemplary embodiment, before training the speech synthesis model to be trained based on the standard-length training data, the method further includes:

[0033] Style extraction is performed on the speech segment to obtain global speech style information of the speech segment;

[0034] The process of training the speech synthesis model based on the standard-length training data includes:

[0035] Based on the standard-length training data and the global speech style information, the speech synthesis model to be trained is trained to obtain a pre-trained speech synthesis model.

[0036] In an exemplary embodiment, the step of extracting style from the speech segment to obtain global speech style information of the speech segment includes:

[0037] Obtain the Mel cepstral spectrum corresponding to the audio segment;

[0038] Style extraction is performed based on the Mel cepstral spectrum to obtain the global speech style information of the speech segment.

[0039] In an exemplary embodiment, the step of performing speech segmentation processing on the paragraph speech using a first model to obtain multiple single-sentence speech includes:

[0040] The first model is used to perform energy recognition on the speech segment, and the silence interval in the speech segment is determined based on the energy recognition result; wherein the energy corresponding to the silence interval is lower than a set energy threshold.

[0041] Based on the silence interval, determine the segmentation position for the speech segment;

[0042] Based on the segmentation position, the speech segment is segmented to obtain the multiple single-sentence speech.

[0043] According to a second aspect of the present disclosure, a speech synthesis model training apparatus is provided, comprising:

[0044] The segmentation unit is configured to acquire unlabeled speech segments, perform speech segmentation processing on the speech segments using a first model, and obtain multiple single-sentence speech segments.

[0045] The conversion unit is configured to acquire the single-sentence text corresponding to each of the multiple single-sentence speech, and perform word-to-phone conversion processing on the single-sentence text through the second model to obtain the single-sentence phoneme sequence corresponding to each of the single-sentence speech;

[0046] The training unit is configured to use the plurality of single-sentence speech and the single-sentence phoneme sequence corresponding to each single-sentence speech as standard-length training data, and train the speech synthesis model to be trained based on the standard-length training data to obtain the trained speech synthesis model.

[0047] In one exemplary embodiment, the apparatus further includes:

[0048] The first augmentation unit is configured to perform concatenation processing on at least two of the plurality of single-sentence speech to obtain corresponding concatenated speech, and to concatenate the single-sentence phoneme sequences corresponding to the at least two single-sentence speech to obtain corresponding concatenated phoneme sequences, and to obtain corresponding long sentence training data based on the concatenated speech and the concatenated phoneme sequences.

[0049] The second augmentation unit is configured to perform segmentation processing on at least one of the plurality of single-sentence speech to obtain corresponding segmented speech, and to segment the single-sentence phoneme sequence corresponding to the at least one single-sentence speech to obtain corresponding segmented phoneme sequence, and to obtain corresponding short sentence training data based on the segmented speech and the segmented phoneme sequence.

[0050] The training unit is further configured to perform training on the speech synthesis model to be trained based on the standard-length training data to obtain a pre-trained speech synthesis model; and to train the pre-trained speech synthesis model based on the standard-length training data, the long sentence training data, and the short sentence training data to obtain a trained speech synthesis model.

[0051] In an exemplary embodiment, the first augmentation unit is further configured to perform silence completion processing and / or pruning processing on the spliced ​​speech to obtain processed spliced ​​speech corresponding to the spliced ​​speech; wherein silence refers to a part of the speech that does not contain the target sound, silence completion processing refers to adding silence to the speech, and silence pruning processing refers to discarding part of the silence in the speech; the processed spliced ​​speech and the spliced ​​phoneme sequence are used as training data for the long sentence.

[0052] In an exemplary embodiment, the first augmentation unit is further configured to perform completion and / or trimming processing on the initial and final silence durations of the spliced ​​speech, so that the initial and final silence durations of the spliced ​​speech are the same; and to perform completion or trimming processing on the silence duration between two adjacent single-sentence speech that make up the spliced ​​speech, so that the silence duration between the two adjacent single-sentence speech is the same as the silence duration between two phonemes in each single-sentence speech.

[0053] In an exemplary embodiment, the second augmentation unit is further configured to input the single-sentence phoneme sequences corresponding to each of the plurality of single-sentence speech into the pre-trained speech synthesis model to obtain alignment information of the single-sentence phoneme sequences; the alignment information characterizes the first and last silence durations of the single-sentence phoneme sequences and the speech length corresponding to each phoneme; according to the alignment information, segmenting at least one of the plurality of single-sentence speech to obtain corresponding segmented speech, and, according to the alignment information, segmenting the single-sentence phoneme sequences corresponding to the at least one single-sentence speech to obtain corresponding segmented phoneme sequences.

[0054] In an exemplary embodiment, the second augmentation unit is further configured to perform silence completion processing and / or pruning processing on the segmented speech to obtain processed segmented speech corresponding to the segmented speech; wherein silence refers to a part of the speech that does not contain the target sound, silence completion processing refers to adding silence to the speech, and silence pruning processing refers to discarding part of the silence in the speech; the processed segmented speech and the segmented phoneme sequence are used as training data for the short sentence.

[0055] In an exemplary embodiment, the second augmentation unit is further configured to perform completion and / or trimming processing on the initial and final silence durations of the segmented speech, so that the initial and final silence durations of the segmented speech are the same.

[0056] In an exemplary embodiment, the training unit is further configured to train the pre-trained speech synthesis model using the short sentence training data to obtain a retrained speech synthesis model; and to train the retrained speech synthesis model using the standard-length training data, the short sentence training data, and the long sentence training data until a fully trained speech synthesis model is obtained.

[0057] In an exemplary embodiment, the training unit is further configured to perform style extraction on the speech segment to obtain global speech style information of the speech segment; and to train the speech synthesis model to be trained based on the standard-length training data and the global speech style information to obtain a pre-trained speech synthesis model.

[0058] In one exemplary embodiment, the training unit further includes a style extraction subunit configured to perform the following: acquiring the Mel cepstral spectrum corresponding to the segment speech; and performing style extraction based on the Mel cepstral spectrum to obtain global speech style information of the segment speech.

[0059] In an exemplary embodiment, the segmentation unit is further configured to perform energy recognition on the speech segment using the first model, determine a silence interval in the speech segment based on the energy recognition result; wherein the energy corresponding to the silence interval is lower than a set energy threshold; determine the segmentation position for the speech segment based on the silence interval; and perform speech segmentation processing on the speech segment based on the segmentation position to obtain the plurality of single-sentence speech.

[0060] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0061] processor;

[0062] Memory used to store the processor's executable instructions;

[0063] The processor is configured to execute the instructions to implement the method as described in any of the preceding methods.

[0064] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in any of the preceding claims.

[0065] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the method as described in any of the preceding claims.

[0066] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0067] The first model performs speech segmentation on unlabeled speech segments to obtain multiple single-sentence speech. After obtaining the corresponding single-sentence text for each single-sentence speech, the second model performs phonetic conversion on the single-sentence text to obtain the single-sentence phoneme sequence for each single-sentence speech. Based on the multiple single-sentence speech and their corresponding single-sentence phoneme sequences, standard-length training data is obtained. The speech synthesis model to be trained is then trained based on the standard-length training data, resulting in a trained speech synthesis model. In this method, the unlabeled speech segments can be recorded in a non-professional environment, thus eliminating the need for a large amount of labeled speech data. In the preparation of training data, the first model automatically segments the unlabeled speech segments, and the human operator only needs to annotate the text of the segmented single-sentence speech. The single-sentence phoneme sequences corresponding to the segmented single-sentence speech can be extracted by the second model, realizing the automated acquisition of single-sentence phoneme sequences and single-sentence speech in the training data. This greatly reduces the dependence on manual labor, shortens the preparation time of training data, and thus improves the training efficiency of the speech synthesis model.

[0068] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0069] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0070] Figure 1 This is a flowchart illustrating a speech synthesis model training method according to an exemplary embodiment.

[0071] Figure 2 This is a schematic diagram of the structure of a Tacotron acoustic model according to an exemplary embodiment.

[0072] Figure 3 This is a schematic diagram of the structure of a VAE-based style encoder according to an exemplary embodiment.

[0073] Figure 4 This is a flowchart illustrating a speech synthesis model training method according to another exemplary embodiment.

[0074] Figure 5 This is a flowchart illustrating a speech synthesis model training method according to an application embodiment.

[0075] Figure 6 This is a structural block diagram of a speech synthesis model training device according to an exemplary embodiment.

[0076] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0077] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0078] It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0079] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0080] In one exemplary embodiment, such as Figure 1 As shown, a speech synthesis model training method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0081] In step S110, unlabeled speech segments are obtained, and the speech segments are segmented using the first model to obtain multiple single-sentence speech segments.

[0082] Among them, the segmented audio can be audio recorded in a low-cost, non-professional environment.

[0083] In a specific implementation, the first model can be a Voice Activation Detection (VAD) model. The VAD model can detect the start point and end point of human voices in a long speech, and thus segment the long speech into multiple short speeches according to the start point and end point. Therefore, after obtaining the unannotated paragraph speech, the start point and end point of human voices in the paragraph speech can be detected through the first model, and then the paragraph speech can be segmented to obtain multiple single-sentence speeches.

[0084] In step S120, the single-sentence text corresponding to each single-sentence speech among the multiple single-sentence speeches is obtained, and the second model is used to perform grapheme-to-phoneme conversion processing on the single-sentence text to obtain the single-sentence phoneme sequence corresponding to each single-sentence speech.

[0085] Among them, the single-sentence text represents the annotation of the single-sentence speech, and can be understood as the speech content expressed by the single-sentence speech. For example, the single-sentence text corresponding to a single-sentence speech is "The weather is very good today".

[0086] Among them, a phoneme is the smallest unit or the smallest speech segment that constitutes a syllable. One pronunciation action forms one phoneme. For example, [ma] contains two pronunciation actions [m] and [a], which are two phonemes. Therefore, the single-sentence phoneme sequence corresponding to a single-sentence speech can be understood as the sequence composed of the phonemes corresponding to all the pronunciation actions in the single-sentence speech. For example, the single-sentence phoneme sequence corresponding to the single-sentence speech "The weather is very good today" is "jin tian tian qi hen hao".

[0087] In a specific implementation, the single-sentence text corresponding to each single-sentence speech can be obtained through annotation. The second model can be a Grapheme-to-Phoneme (G2P) model. The G2P model usually adopts a rule-based dictionary mapping method to output the phonemes of the text. Therefore, after obtaining the single-sentence text corresponding to each single-sentence speech, each single-sentence text can be input into the second model respectively. The second model uses the dictionary mapping method to perform grapheme-to-phoneme conversion processing on each single-sentence text, and outputs the phonemes corresponding to each single-sentence text. For each single-sentence text, all the phonemes in the single-sentence text are formed into a phoneme sequence in the speech order, which is correspondingly used as the single-sentence phoneme sequence corresponding to each single-sentence speech.

[0088] In step S130, the multiple single-sentence speeches and the single-sentence phoneme sequence corresponding to each single-sentence speech are used as standard-length training data, and the speech synthesis model to be trained is trained based on the standard-length training data to obtain the trained speech synthesis model.

[0089] In practice, the training method for the speech synthesis model using standard-length training data is as follows: the single-sentence phoneme sequence corresponding to each single-sentence speech in the standard-length training data is used as the input variable, and each single-sentence speech is used as the supervision information to train the speech synthesis model to be trained.

[0090] More specifically, using a single-sentence phoneme sequence as the input variable, the speech synthesis model to be trained is input, and the speech synthesis model outputs predicted synthesized speech. By determining the loss value between the predicted synthesized speech and the actual single-sentence speech, the training ends when the loss value converges to a preset accuracy or reaches a preset number of training iterations. The speech synthesis model to be trained is then trained to obtain the trained speech synthesis model.

[0091] In practical applications, the speech synthesis model used in this disclosure can be the Tacotron acoustic model, for reference. Figure 2 This is a schematic diagram of the Tacotron acoustic model. The Tacotron acoustic model is based on the Seq2Seq model (a model where the input and output sequences are not aligned). It uses an autoregressive decoder to predict acoustic feature sequences frame by frame and employs an attention mechanism with phonemes as input variables. The model includes a word embedding layer, a preprocessing network, and an Encoder-Decoder structure. After outputting the original spectrum, a lightweight neural network vocoder is used to generate the speech waveform.

[0092] In the aforementioned speech synthesis model training method, a first model performs speech segmentation on unlabeled speech segments to obtain multiple single-sentence speech. After obtaining the corresponding single-sentence text for each single-sentence speech, a second model performs phonetic conversion on the single-sentence text to obtain the single-sentence phoneme sequence corresponding to each single-sentence speech. Based on the multiple single-sentence speech and their corresponding single-sentence phoneme sequences, standard-length training data is obtained. The speech synthesis model to be trained is then trained based on this standard-length training data, resulting in a trained speech synthesis model. In this method, the unlabeled speech segments can be recorded in a non-professional environment, thus eliminating the need for a large amount of labeled speech data. During the preparation of training data, the first model automatically segments the unlabeled speech segments, and manual annotation of the segmented single-sentence speech is required. Furthermore, the single-sentence phoneme sequences corresponding to the segmented single-sentence speech can be extracted by the second model, achieving automated acquisition of single-sentence phoneme sequences and single-sentence speech in the training data. This significantly reduces reliance on manual labor, shortens the preparation time of training data, and thus improves the training efficiency of the speech synthesis model.

[0093] In one exemplary embodiment, the above-described speech synthesis model training method further includes:

[0094] Step S210: At least two single-sentence speech samples from multiple single-sentence speech samples are spliced ​​together to obtain the corresponding spliced ​​speech sample; and the single-sentence phoneme sequences corresponding to at least two single-sentence speech samples are spliced ​​together to obtain the corresponding spliced ​​phoneme sequence; and the corresponding long sentence training data is obtained based on the spliced ​​speech sample and the spliced ​​phoneme sequence.

[0095] Step S220: Segment at least one of the multiple single-sentence speech segments to obtain the corresponding segmented speech; and segment the single-sentence phoneme sequence corresponding to at least one single-sentence speech segment to obtain the corresponding segmented phoneme sequence; and obtain the corresponding short sentence training data based on the segmented speech and the segmented phoneme sequence.

[0096] Step S230: Train the speech synthesis model to be trained based on standard length training data to obtain a pre-trained speech synthesis model.

[0097] Step S240: Based on standard-length training data, long sentence training data, and short sentence training data, train the pre-trained speech synthesis model to obtain the trained speech synthesis model.

[0098] In specific implementation, considering the impact of the amount of speech training data on the training results of the speech synthesis model, this disclosure also proposes a data augmentation scheme for standard-length training data, namely, concatenating standard-length training data to obtain corresponding long sentence training data, and segmenting standard-length training data to obtain corresponding short sentence training data, and using standard-length training data, long sentence training data and short sentence training data together as training data for training the speech synthesis model.

[0099] More specifically, the long sentence training data is obtained by randomly concatenating single-sentence speech and single-sentence phoneme sequences from the standard-length training data. Specifically, at least two single-sentence speech samples from the standard-length training data are concatenated to obtain the corresponding concatenated speech, and the single-sentence phoneme sequences corresponding to at least two single-sentence speech samples are concatenated to obtain the corresponding concatenated phoneme sequences. Based on the concatenated speech and the concatenated phoneme sequences, the corresponding long sentence training data is obtained. A single speech sample may not participate in the concatenation or may participate in multiple concatenations; the goal is simply to obtain some long sentences, but each concatenation requires at least two single-sentence speech samples.

[0100] The method for obtaining short sentence training data is as follows: segmenting the single-sentence speech and single-sentence phoneme sequence in the standard-length training data, specifically, segmenting at least one single-sentence speech in the standard-length training data to obtain the corresponding segmented speech, and segmenting the single-sentence phoneme sequence corresponding to at least one single-sentence speech to obtain the corresponding segmented phoneme sequence, and obtaining the corresponding short sentence training data based on the segmented speech and segmented phoneme sequence.

[0101] After obtaining short sentence training data and long sentence training data, the speech synthesis model pre-trained with standard length training data, long sentence training data and short sentence training data are used to retrain the speech synthesis model until the trained speech synthesis model is obtained.

[0102] In this embodiment, long sentence training data is obtained by concatenating standard-length training data, and short sentence training data is obtained by segmenting standard-length training data. This augments the standard-length training data, increases the amount of training data, and thus improves the training effect of the speech synthesis model. Specifically, long sentence training data can be used to improve the stability of the synthesized speech of the speech synthesis model, and short sentence training data can be used to improve the expressiveness of the synthesized speech of the speech synthesis model.

[0103] In an exemplary embodiment, the step S210 above, which obtains the corresponding long sentence training data based on the concatenated speech and the concatenated phoneme sequence, includes:

[0104] Step S211: Perform silence completion processing and / or trimming processing on the spliced ​​speech to obtain the processed spliced ​​speech corresponding to the spliced ​​speech; wherein, silence means the part of the speech that does not contain the target sound, silence completion processing means adding silence to the speech, and silence trimming processing means discarding part of the silence in the speech.

[0105] Step S212: Use the processed concatenated speech and concatenated phoneme sequence as training data for long sentences.

[0106] Among them, the silence completion process means adding a silence of a certain duration between two speech sounds.

[0107] Among them, the silencing trimming process means reducing the duration of silence between two speech words.

[0108] In practice, after concatenating two single-sentence speech, the silence duration at the beginning and end of the concatenated speech, as well as the silence duration at the concatenation position, needs to be processed to avoid excessively long or short silence durations that could affect the rationality and effectiveness of the concatenated speech. Therefore, silence completion and / or pruning processing is required on the obtained concatenated speech. The processed concatenated speech and the concatenated phoneme sequence are then used to construct long sentence training data.

[0109] Further, in an exemplary embodiment, step S211 involves performing silence completion and / or trimming processing on the spliced ​​speech to obtain the processed spliced ​​speech, including: performing silence completion and / or trimming processing on the initial and final silence durations of the spliced ​​speech to make the initial and final silence durations of the spliced ​​speech the same; and performing silence completion or trimming processing on the silence duration between two adjacent single-sentence speech that make up the spliced ​​speech to make the silence duration between two adjacent single-sentence speech the same as the silence duration between two phonemes in each single-sentence speech.

[0110] In practical implementation, the processing of the silence duration at the beginning and end of the concatenated speech can be based on the condition that the silence duration at the beginning and end of the concatenated speech is the same. This involves supplementing and / or pruning the silence duration at the beginning and end of the concatenated speech. More specifically, a fixed duration can be preset. After determining the silence duration at the beginning and end of the concatenated speech, silence durations exceeding the fixed duration are pruned, and silence durations shorter than the fixed duration are supplemented, thereby achieving the same silence duration at the beginning and end of the concatenated speech.

[0111] The processing of silence duration at the splicing position of concatenated speech can be based on the condition that the silence duration at the splicing position is the same as the silence duration between two adjacent single-sentence speech units in each single-sentence speech unit. This condition allows for the completion and / or trimming of the silence duration at the splicing position. More specifically, if the silence duration at the splicing position exceeds the silence duration between two phonemes in a single-sentence speech unit, then the silence duration at the splicing position is trimmed; otherwise, it is completed.

[0112] It's important to note that phoneme sequences do not require silence completion or clipping. Silence can be understood as the duration of a pause between two sentences or two words during the speaking process; the presence or absence of a pause affects the final sound, thus requiring silence completion or clipping. However, phoneme sequences are textual information, such as the pinyin of a sentence, like "wo shiyi ge xue sheng." Regardless of whether there are pauses during the speaking process or the duration of those pauses, the phoneme sequence remains unchanged, therefore requiring no silence completion or clipping.

[0113] In this embodiment, the silence duration at the beginning and end of the spliced ​​speech, as well as the silence duration at the splicing position, are supplemented or trimmed to ensure the rationality and effectiveness of the spliced ​​speech, thereby ensuring the effective training of the speech synthesis model with long sentence training data.

[0114] In an exemplary embodiment, step S220 above, which involves segmenting at least one single-sentence speech from a plurality of single-sentence speech to obtain corresponding segmented speech, and segmenting the single-sentence phoneme sequence corresponding to at least one single-sentence speech to obtain a corresponding segmented phoneme sequence, includes:

[0115] Step S221: Input the single-sentence phoneme sequences corresponding to each of the multiple single-sentence speech into the pre-trained speech synthesis model to obtain the alignment information of the single-sentence phoneme sequences; the alignment information represents the first and last silence durations of the single-sentence phoneme sequences and the speech length corresponding to each phoneme;

[0116] Step S222: Based on the alignment information, at least one single-sentence speech in a plurality of single-sentence speech is segmented to obtain the corresponding segmented speech; and based on the alignment information, the single-sentence phoneme sequence corresponding to at least one single-sentence speech is segmented to obtain the corresponding segmented phoneme sequence.

[0117] In the specific implementation, the standard-length training data is segmented to obtain short-sentence training data. The segmentation basis is the alignment information of the single-sentence phoneme sequences in the standard-length training data, namely the silence duration at the beginning and end of the single-sentence phoneme sequence and the speech length corresponding to each phoneme. The alignment information of the single-sentence phoneme sequences can be obtained by inputting the single-sentence phoneme sequences corresponding to each single-sentence speech in the standard-length training data into a pre-trained speech synthesis model. The pre-trained speech synthesis model processes the synthesized speech, and based on the speech length and silence duration of the synthesized speech, the silence duration at the beginning and end of the single-sentence phoneme sequence and the speech length corresponding to each phoneme are obtained. Further, the silence duration at the beginning and end of the single-sentence phoneme sequence and the speech length corresponding to each phoneme are used as alignment information. Based on the alignment information, the segmentation position is determined. According to the condition that the short-sentence length is greater than a first set value and the silence duration at the segmentation position is greater than a second set value, at least one single-sentence speech among multiple single-sentence speech is segmented to obtain the corresponding segmented speech. In addition, the single-sentence phoneme sequence corresponding to at least one single-sentence speech is segmented to obtain the corresponding segmented phoneme sequence.

[0118] In this embodiment, the standard-length training data is segmented into short sentences using alignment information extracted from a pre-trained speech synthesis model. This ensures the rationality and accuracy of the selected segmentation positions, thereby improving the effectiveness of the obtained short sentence training data.

[0119] In an exemplary embodiment, the step S220 above, which obtains corresponding short sentence training data based on segmented speech and segmented phoneme sequences, includes:

[0120] Step S223: Perform silence completion processing and / or pruning processing on the segmented speech to obtain the processed segmented speech corresponding to the segmented speech; wherein, silence refers to the part of the speech that does not contain the target sound, silence completion processing means adding silence to the speech, and silence pruning processing means discarding part of the silence in the speech.

[0121] Step S224: The processed segmented speech and segmented phoneme sequence are used as training data for short sentences.

[0122] In practice, after segmenting a single sentence of speech, the silence duration at the beginning and end of the segmented speech needs to be processed accordingly to avoid excessively long or short silence durations that could affect the rationality and effectiveness of the segmentation. Therefore, it is necessary to perform silence completion and / or pruning processing on the obtained segmented speech, and then combine the processed segmented speech with the segmented phoneme sequence to form long sentence training data.

[0123] Furthermore, in an exemplary embodiment, step S223 involves performing silence completion and / or trimming processing on the segmented speech to obtain the processed segmented speech, including: performing silence completion and / or trimming processing on the silence duration at the beginning and end of the segmented speech to make the silence duration at the beginning and end of the segmented speech the same.

[0124] In practical implementation, the processing of the silence duration at the beginning and end of segmented speech can be based on the condition that the silence duration at the beginning and end of the segmented speech is the same. This involves supplementing and / or pruning the silence duration at the beginning and end of the segmented speech. More specifically, a fixed duration can be preset. After determining the silence duration at the beginning and end of the segmented speech, silence durations exceeding the fixed duration are pruned, and silence durations shorter than the fixed duration are supplemented, thereby achieving the same silence duration at the beginning and end of the segmented speech.

[0125] In this embodiment, the silence duration at the beginning and end of the segmented speech is supplemented or trimmed to ensure the rationality and effectiveness of the segmented speech, thereby ensuring that the short sentence training data can effectively train the expressiveness of the speech synthesis model.

[0126] In an exemplary embodiment, step S230 above, which trains a pre-trained speech synthesis model based on standard-length training data, long-sentence training data, and short-sentence training data to obtain a trained speech synthesis model, includes:

[0127] Step S231: Train the pre-trained speech synthesis model using short sentence training data to obtain a retrained speech synthesis model.

[0128] Step S232: Train the retrained speech synthesis model using standard-length training data, short-sentence training data, and long-sentence training data until a fully trained speech synthesis model is obtained.

[0129] In practice, the lower the quality of the training data, the lower the sound quality of the synthesized speech. To address this issue, this embodiment proposes a phased training method. Since the speech synthesis model uses an autoregressive model, and considering the characteristics of autoregressive models, which are easier to model short sequence data, this embodiment first trains the pre-trained speech synthesis model using short sentence training data. For long sequence data, which is more difficult to model, adaptive training is performed based on the speech synthesis model trained on the short sentence training data. Specifically, standard-length training data, long sentence training data, and short sentence training data can be combined to construct training data of mixed duration. This mixed-duration training data is used to train the speech synthesis model until a fully trained speech synthesis model is obtained.

[0130] In this embodiment, the speech synthesis model is first trained using short sentence training data, which is easier to model. Then, a mixed-length training data set, constructed from standard-length training data, short sentence training data, and long sentence training data, is used to further train the speech synthesis model trained on the short sentence training data. This reduces the impact of training data quality on the sound quality of the synthesized speech, effectively improving the sound quality performance of the synthesized speech. Furthermore, due to the characteristic of autoregressive models using themselves as regression variables, short sentence training data can effectively improve the modeling ability of the decoder module in the speech synthesis model, thereby increasing the stability of the synthesized speech. Conversely, long sentence training data can effectively improve the modeling ability of the encoder and attention mechanism modules, thereby enhancing the expressiveness of the synthesized speech.

[0131] In an exemplary embodiment, before training the speech synthesis model to be trained based on standard-length training data in step S130, the method further includes: extracting the style of the speech segment to obtain global speech style information of the speech segment;

[0132] Training a speech synthesis model to be trained based on standard-length training data includes: training the speech synthesis model to be trained based on standard-length training data and global speech style information to obtain a pre-trained speech synthesis model.

[0133] Specifically, since the same speech content can have different speech styles, in order to avoid the variability of the speech style of the synthesized speech output by the speech synthesis model, the global speech style information of the recorded speech segments can be extracted in advance. The global speech style information and standard length training data are used as input variables and input into the speech synthesis model to be trained to obtain the pre-trained speech synthesis model.

[0134] Furthermore, in an exemplary embodiment, style extraction is performed on the speech segment to obtain global speech style information of the speech segment, including: obtaining the Mel cepstrum corresponding to the speech segment; and performing style extraction based on the Mel cepstrum to obtain global speech style information of the speech segment.

[0135] In practice, audio feature extraction processing can be performed on the unlabeled speech segments to obtain the Mel cepstrum (MEL) of the unlabeled speech segments. Then, a style encoder based on VAE (Variational Autoencoder) can be used to perform global style encoding on the Mel cepstrum to obtain the global speech style information of the speech segments.

[0136] More specifically, the VAE-based low-noise control method works as follows: This style coding model obtains a continuous and complete style latent space distribution through the VAE structure, thereby improving style control capabilities. For example... Figure 3 The diagram shows the structure of a VAE-based style encoder, including a feature compression and excitation residual network (SE ResNet). Input variables are passed through six convolutions (6*Conv2D) and then fed into an average pooling layer. They then pass through one linear layer, two nonlinear layers, a classification layer, and a scaling layer before entering a gated recurrent unit (GRU) layer. After processing by the GRU layer, the inputs are fed into a mean-linear layer and a variance-linear layer, respectively. Reparameterization is then performed based on the output mean and variance. In this process, the style transformation layer inputs the last state value of the GRU layer (gated recurrent unit) into two fully connected layers (mean-linear layer and variance-linear layer) to obtain the mean and variance of a multivariate Gaussian distribution. Finally, a 64-dimensional vector is sampled from this Gaussian distribution as the input to the decoder (connected to the Pre-Net output in each step). Since backpropagation cannot handle random operations in the network, VAE introduces a "reparameterization technique." During training, the KL loss can easily drop to zero, a phenomenon known as KL collapse. Three solutions are proposed to address this problem: 1) Introducing KL annealing, where the weights of the KL loss gradually increase; 2) Employing a phased optimization approach, first optimizing the reconstruction loss, then optimizing the KL loss; 3) Limiting the minimum value of the KL loss by introducing a minimum hyperparameter. This process can be expressed by the following formula: in, Indicates KL loss, D KL [] represents the KL distance between the predicted distribution and the Gaussian distribution. Indicates a Gaussian distribution. Let represent the mean and variance of the prediction distribution, respectively, and Δ represent the minimum hyperparameter used to limit the KL loss. Furthermore, scaling the global style embedding can effectively control the style intensity of the synthesized speech.

[0137] Understandably, since unlabeled audio segments can be recorded in non-professional environments, they will contain noise. If style extraction is performed directly on these segments, the results will be very poor. However, extracting MEL features can extract the effective information from the audio segments and filter out the useless information. The principle is to simulate the structure of the human ear and filter the audio. The processed data is then used for style extraction, which will significantly improve the results.

[0138] In this embodiment, the Mel cepstral spectrum is selected as the basis for global style coding. The high expressiveness and low background noise of the Mel cepstral spectrum can effectively solve the problem of low background noise in the speech segment.

[0139] In an exemplary embodiment, step S110 involves performing speech segmentation processing on the segmented speech using a first model to obtain multiple single-sentence speech, including:

[0140] Step S111: Energy recognition is performed on the speech segment using the first model, and the silence interval in the speech segment is determined based on the energy recognition result; wherein, the energy corresponding to the silence interval is lower than a set energy threshold.

[0141] Step S112: Determine the segmentation position for the speech segment based on the silence interval;

[0142] Step S113: Perform speech segmentation processing on the speech segment based on the segmentation position to obtain multiple single-sentence speech.

[0143] In practical implementation, the first model can perform energy recognition on the speech segment to identify the silence intervals. Based on the silence intervals, the segmentation position for the speech segment is determined. Further, based on the segmentation position, the speech segment is segmented to obtain multiple single-sentence speech. More specifically, speech segments with silence intervals greater than a first threshold and non-silent intervals greater than a second threshold can be segmented to extract clauses, thus obtaining multiple single-sentence speech. The second threshold is greater than the first threshold, meaning the threshold for non-silent intervals is greater than the threshold for silent intervals. This process is fully automated and extremely fast. Furthermore, for the unlabeled clauses after segmentation, only low-cost manual text annotation is required, using a trained G2P (text-to-phoneme) model to extract phoneme sequences. The accuracy of the polyphonic character model in this scheme can reach 91.47%, providing high-accuracy phoneme sequence labels.

[0144] The automatic clause segmentation method based on VAD provided in this embodiment can overcome the shortcomings of traditional methods, which require manual segmentation of clauses by annotators after the speaker has recorded a complete paragraph of speech, and phoneme sequence annotation of the clauses. This requires training of standard personnel, is very time-consuming, and has a high annotation error rate.

[0145] In another exemplary embodiment, such as Figure 4 The diagram shown is a flowchart illustrating another speech synthesis model training method according to an exemplary embodiment. In this embodiment, the method includes the following steps:

[0146] Step S410: Obtain unlabeled speech segments, perform energy recognition on the speech segments using the first model, and determine the silence intervals in the speech segments based on the energy recognition results; wherein, the energy corresponding to the silence intervals is lower than a set energy threshold.

[0147] Step S420: Determine the segmentation position for the speech segment based on the silence interval;

[0148] Step S430: Perform speech segmentation processing on the speech segment based on the segmentation position to obtain multiple single-sentence speech;

[0149] Step S440: Obtain the single-sentence text corresponding to each single-sentence speech, and perform word-to-phonetic conversion processing on the single-sentence text through the second model to obtain the single-sentence phoneme sequence corresponding to each single-sentence speech.

[0150] Step S450: Take multiple single-sentence speech and the single-sentence phoneme sequence corresponding to each single-sentence speech as standard-length training data, train the speech synthesis model to be trained based on the standard-length training data, and obtain the pre-trained speech synthesis model.

[0151] Step S460: At least two single-sentence speech samples from a plurality of single-sentence speech samples are spliced ​​together to obtain the corresponding spliced ​​speech sample; and the single-sentence phoneme sequences corresponding to at least two single-sentence speech samples are spliced ​​together to obtain the corresponding spliced ​​phoneme sequence.

[0152] Step S461: Perform silence completion processing and / or trimming processing on the spliced ​​speech to obtain the processed spliced ​​speech corresponding to the spliced ​​speech;

[0153] Step S462: Based on the processed concatenated speech and the concatenated phoneme sequence, long sentence training data is obtained;

[0154] Step S470: Input the single-sentence phoneme sequences corresponding to each of the multiple single-sentence speech into the pre-trained speech synthesis model to obtain the alignment information of the single-sentence phoneme sequences; the alignment information represents the first and last silence durations of the single-sentence phoneme sequences and the speech length corresponding to each phoneme;

[0155] Step S471: Based on the alignment information, at least one single-sentence speech in a plurality of single-sentence speech is segmented to obtain the corresponding segmented speech; and based on the alignment information, the single-sentence phoneme sequence corresponding to at least one single-sentence speech is segmented to obtain the corresponding segmented phoneme sequence.

[0156] Step S472: Perform silence completion and / or trimming on the segmented speech to obtain the processed segmented speech.

[0157] Step S473: Based on the processed segmented speech and segmented phoneme sequence, short sentence training data is obtained;

[0158] Step S474: Train the pre-trained speech synthesis model using short sentence training data to obtain a retrained speech synthesis model.

[0159] Step S480: Train the retrained speech synthesis model using standard-length training data, short-sentence training data, and long-sentence training data until a fully trained speech synthesis model is obtained.

[0160] In one exemplary embodiment, to facilitate understanding of the embodiments of this disclosure by those skilled in the art, specific examples will be described below in conjunction with the accompanying drawings. References Figure 5 This is a flowchart illustrating the training method for a speech synthesis model in an application example, specifically including the following steps:

[0161] 1. Input unlabeled speech segments and use the VAD (Voice Activation Detection) model to perform energy detection and segmentation to obtain unlabeled single-sentence speech.

[0162] 2. Unannotated single-sentence speech is manually annotated to obtain single-sentence text.

[0163] 3. Single-sentence texts are processed using a G2P model (Grapheme-to-Phoneme) to obtain single-sentence phoneme sequences.

[0164] 4. Construct standard-length training data from unlabeled single-sentence speech and single-sentence phoneme sequences.

[0165] 5. The speech synthesis model Tacotron is pre-trained using standard-length training data, and the alignment information of the standard-length training data is obtained through the pre-trained Tacotron.

[0166] 6. Based on the alignment information, the standard-length training data is segmented and silenced to obtain short sentence training data (short sentence phoneme sequence + short sentence speech).

[0167] 7. Randomly concatenate and mute the standard-length training data to obtain long-sentence training data (long-sentence phoneme sequence + long-sentence speech).

[0168] 8. Retrain the pre-trained Tacotron with short-duration corpora using short sentence training data.

[0169] 9. By using standard-length training data, short-sentence training data, and long-sentence training data, adaptive training of Tacotron mixed-length corpus is carried out on the basis of the Tacotron short-duration corpus training model to obtain the final speech synthesis model.

[0170] This embodiment provides an efficient and low-cost speech synthesis timbre expansion scheme. It can quickly construct training corpora using low-cost, unlabeled speech in non-professional scenarios. Through various training strategies and data augmentation methods, it can improve the sound quality, stability, expressiveness, and naturalness of low-cost acoustic models, which can greatly save economic and time costs.

[0171] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0172] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0173] Based on the same inventive concept, this disclosure also provides a speech synthesis model training apparatus for implementing the speech synthesis model training method described above.

[0174] Figure 6 This is a structural block diagram illustrating a speech synthesis model training device according to an exemplary embodiment. (Refer to...) Figure 6 The device includes:

[0175] The segmentation unit 610 is configured to acquire unlabeled speech segments, perform speech segmentation processing on the speech segments using the first model, and obtain multiple single-sentence speech segments.

[0176] The conversion unit 620 is configured to acquire the single-sentence text corresponding to each single-sentence speech in multiple single-sentence speech, and perform word-to-phone conversion processing on the single-sentence text through the second model to obtain the single-sentence phoneme sequence corresponding to each single-sentence speech.

[0177] Training unit 630 is configured to use the plurality of single-sentence speech and the single-sentence phoneme sequence corresponding to each single-sentence speech as standard-length training data, and train the speech synthesis model to be trained based on the standard-length training data to obtain the trained speech synthesis model.

[0178] In one exemplary embodiment, the apparatus further includes:

[0179] The first augmentation unit is configured to perform concatenation processing on at least two single-sentence speech samples from a plurality of single-sentence speech samples to obtain corresponding concatenated speech samples, and to concatenate single-sentence phoneme sequences corresponding to at least two single-sentence speech samples to obtain corresponding concatenated phoneme sequences, and to obtain corresponding long sentence training data based on the concatenated speech samples and the concatenated phoneme sequences.

[0180] The second augmenting unit is configured to perform segmentation processing on at least one of a plurality of single-sentence speech to obtain the corresponding segmented speech, and to segment the single-sentence phoneme sequence corresponding to at least one single-sentence speech to obtain the corresponding segmented phoneme sequence, and to obtain the corresponding short sentence training data based on the segmented speech and the segmented phoneme sequence.

[0181] The training unit 630 is further configured to perform training on the speech synthesis model to be trained based on the standard-length training data to obtain a pre-trained speech synthesis model; and to train the pre-trained speech synthesis model based on the standard-length training data, the long sentence training data, and the short sentence training data to obtain a trained speech synthesis model.

[0182] In an exemplary embodiment, the first augmentation unit is further configured to perform silence completion processing and / or pruning processing on the spliced ​​speech to obtain the processed spliced ​​speech corresponding to the spliced ​​speech; wherein silence refers to the part of the speech that does not contain the target sound, silence completion processing refers to adding silence to the speech, and silence pruning processing refers to discarding part of the silence in the speech; the processed spliced ​​speech and the spliced ​​phoneme sequence are used as training data for long sentences.

[0183] In an exemplary embodiment, the first augmentation unit is further configured to perform completion and / or trimming processing on the initial and final silence durations of the concatenated speech, so that the initial and final silence durations of the concatenated speech are the same; and to perform completion or trimming processing on the silence duration between two adjacent single-sentence speech that make up the concatenated speech, so that the silence duration between two adjacent single-sentence speech is the same as the silence duration between two phonemes in each single-sentence speech.

[0184] In an exemplary embodiment, the second augmentation unit is further configured to input the single-sentence phoneme sequences corresponding to each of the plurality of single-sentence speech into a pre-trained speech synthesis model to obtain alignment information of the single-sentence phoneme sequences; the alignment information characterizes the first and last silence durations of the single-sentence phoneme sequences and the speech length corresponding to each phoneme; based on the alignment information, segmenting at least one single-sentence speech among the plurality of single-sentence speech to obtain corresponding segmented speech, and segmenting the single-sentence phoneme sequences corresponding to at least one single-sentence speech to obtain corresponding segmented phoneme sequences based on the alignment information.

[0185] In an exemplary embodiment, the second augmentation unit is further configured to perform silence completion processing and / or pruning processing on the segmented speech to obtain processed segmented speech corresponding to the segmented speech; wherein silence refers to a part of the speech that does not contain the target sound, silence completion processing refers to adding silence to the speech, and silence pruning processing refers to discarding part of the silence in the speech; the processed segmented speech and the segmented phoneme sequence are used as short sentence training data.

[0186] In one exemplary embodiment, the second augmentation unit is further configured to perform completion and / or trimming processing on the initial and final silence durations of the segmented speech, so that the initial and final silence durations of the segmented speech are the same.

[0187] In one exemplary embodiment, the training unit 630 is further configured to perform training on the pre-trained speech synthesis model using short sentence training data to obtain a retrained speech synthesis model; and to train the retrained speech synthesis model using standard-length training data, short sentence training data, and long sentence training data until a fully trained speech synthesis model is obtained.

[0188] In an exemplary embodiment, the training unit 630 is further configured to perform style extraction on the speech segment to obtain global speech style information of the speech segment; and to train the speech synthesis model to be trained based on standard-length training data and the global speech style information to obtain a pre-trained speech synthesis model.

[0189] In one exemplary embodiment, the training unit 630 further includes a style extraction subunit configured to perform the following: acquiring the Mel cepstrum corresponding to the segment speech; performing style extraction based on the Mel cepstrum to obtain global speech style information of the segment speech.

[0190] In an exemplary embodiment, the segmentation unit 610 is further configured to perform energy recognition on the speech segment using a first model, determine a silence interval in the speech segment based on the energy recognition result, wherein the energy corresponding to the silence interval is lower than a set energy threshold; determine the segmentation position for the speech segment based on the silence interval; and perform speech segmentation processing on the speech segment based on the segmentation position to obtain multiple single-sentence speech.

[0191] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0192] Figure 7 This is a block diagram illustrating an electronic device 700 for implementing a speech synthesis model training method according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0193] Reference Figure 7 The electronic device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output (I / O) interface 712, sensor component 714, and communication component 716.

[0194] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0195] Memory 704 is configured to store various types of data to support the operation of electronic device 700. Examples of such data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene memory.

[0196] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0197] Multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0198] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0199] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0200] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 can detect the on / off state of electronic device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or its components, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0201] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0202] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0203] In one exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0204] In one exemplary embodiment, a computer program product is also provided, which includes instructions that can be executed by a processor 720 of an electronic device 700 to perform the above-described method.

[0205] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0206] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0207] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training a speech synthesis model, characterized in that, include: Unlabeled speech segments are obtained, and the speech segments are segmented using a first model to obtain multiple single-sentence speech segments. The single-sentence text corresponding to each of the multiple single-sentence speech is obtained, and the single-sentence text is processed by the second model to obtain the single-sentence phoneme sequence corresponding to each single-sentence speech. The multiple single-sentence speech and the corresponding single-sentence phoneme sequences are used as standard-length training data. Data augmentation is performed on the standard-length training data to obtain long-sentence training data and short-sentence training data. The long-sentence training data is obtained by concatenating the standard-length training data, and the short-sentence training data is obtained by segmenting the standard-length training data. The speech synthesis model to be trained is trained based on the standard length training data to obtain a pre-trained speech synthesis model. Based on the standard-length training data, the long-sentence training data, and the short-sentence training data, the pre-trained speech synthesis model is trained to obtain the trained speech synthesis model.

2. The method according to claim 1, characterized in that, The process of augmenting the standard-length training data to obtain long-sentence training data and short-sentence training data includes: At least two single-sentence speech samples from the plurality of single-sentence speech samples in the standard-length training data are spliced ​​together to obtain the corresponding spliced ​​speech sample. The single-sentence phoneme sequences corresponding to the at least two single-sentence speech samples are spliced ​​together to obtain the corresponding spliced ​​phoneme sequence. The corresponding long sentence training data is obtained based on the spliced ​​speech sample and the spliced ​​phoneme sequence. At least one single-sentence speech in the standard-length training data is segmented to obtain a corresponding segmented speech. The single-sentence phoneme sequence corresponding to the at least one single-sentence speech is segmented to obtain a corresponding segmented phoneme sequence. Based on the segmented speech and the segmented phoneme sequence, corresponding short sentence training data is obtained.

3. The method according to claim 2, characterized in that, The process of obtaining the corresponding long sentence training data based on the spliced ​​speech and the spliced ​​phoneme sequence includes: The spliced ​​speech is subjected to silence completion processing and / or pruning processing to obtain the processed spliced ​​speech corresponding to the spliced ​​speech; wherein, silence refers to the part of the speech that does not contain the target sound, silence completion processing means adding silence to the speech, and silence pruning processing means discarding part of the silence in the speech. The processed and spliced ​​speech and the spliced ​​phoneme sequence are used as training data for the long sentence.

4. The method according to claim 3, characterized in that, The step of performing silence completion and / or trimming on the spliced ​​speech to obtain the processed spliced ​​speech includes: The initial and final silence durations of the spliced ​​speech are supplemented and / or trimmed to make the initial and final silence durations of the spliced ​​speech the same. Furthermore, the silence duration between two adjacent single-sentence speech units that make up the spliced ​​speech is supplemented or trimmed so that the silence duration between the two adjacent single-sentence speech units is the same as the silence duration between two phonemes in each single-sentence speech unit.

5. The method according to claim 2, characterized in that, The step of segmenting at least one of the plurality of single-sentence speech to obtain the corresponding segmented speech, and segmenting the single-sentence phoneme sequence corresponding to the at least one single-sentence speech to obtain the corresponding segmented phoneme sequence, includes: The single-sentence phoneme sequences corresponding to each of the multiple single-sentence speech are input into the pre-trained speech synthesis model to obtain the alignment information of the single-sentence phoneme sequences; the alignment information represents the first and last silence durations of the single-sentence phoneme sequences and the speech length corresponding to each phoneme; Based on the alignment information, at least one of the multiple single-sentence speech segments is segmented to obtain the corresponding segmented speech; and based on the alignment information, the single-sentence phoneme sequence corresponding to the at least one single-sentence speech segment is segmented to obtain the corresponding segmented phoneme sequence.

6. The method according to claim 2, characterized in that, The process of obtaining the corresponding short sentence training data based on the segmented speech and the segmented phoneme sequence includes: The segmented speech is subjected to silence completion processing and / or pruning processing to obtain the processed segmented speech corresponding to the segmented speech; wherein, silence refers to the part of the speech that does not contain the target sound, silence completion processing means adding silence to the speech, and silence pruning processing means discarding part of the silence in the speech. The processed segmented speech and the segmented phoneme sequence are used as training data for the short sentence.

7. The method according to claim 6, characterized in that, The step of performing silence completion and / or cropping on the segmented speech to obtain the processed segmented speech includes: The silence duration at the beginning and end of the segmented speech is completed and / or trimmed to make the silence duration at the beginning and end of the segmented speech the same.

8. The method according to claim 2, characterized in that, The step of training the pre-trained speech synthesis model based on the standard-length training data, the long sentence training data, and the short sentence training data to obtain the trained speech synthesis model includes: The pre-trained speech synthesis model is trained using the short sentence training data to obtain a retrained speech synthesis model. The retrained speech synthesis model is trained using the standard-length training data, the short-sentence training data, and the long-sentence training data until a fully trained speech synthesis model is obtained.

9. The method according to claim 2, characterized in that, Before training the speech synthesis model to be trained based on the standard-length training data, the process also includes: Style extraction is performed on the speech segment to obtain global speech style information of the speech segment; The process of training the speech synthesis model based on the standard-length training data includes: Based on the standard-length training data and the global speech style information, the speech synthesis model to be trained is trained to obtain a pre-trained speech synthesis model.

10. The method according to any one of claims 1 to 9, characterized in that, The first model is used to segment the speech of the paragraph to obtain multiple single-sentence speech, including: The first model is used to perform energy recognition on the speech segment, and the silence interval in the speech segment is determined based on the energy recognition result; wherein the energy corresponding to the silence interval is lower than a set energy threshold. Based on the silence interval, determine the segmentation position for the speech segment; Based on the segmentation position, the speech segment is segmented to obtain the multiple single-sentence speech.

11. A speech synthesis model training device, characterized in that, include: The segmentation unit is configured to acquire unlabeled speech segments, perform speech segmentation processing on the speech segments using a first model, and obtain multiple single-sentence speech segments. The conversion unit is configured to acquire the single-sentence text corresponding to each of the multiple single-sentence speech, and perform word-to-phone conversion processing on the single-sentence text through the second model to obtain the single-sentence phoneme sequence corresponding to each of the single-sentence speech; The training unit is configured to perform data augmentation on the standard-length training data, using the plurality of single-sentence speech and the corresponding single-sentence phoneme sequences as standard-length training data, to obtain long-sentence training data and short-sentence training data; train a speech synthesis model to be trained based on the standard-length training data to obtain a pre-trained speech synthesis model; and train the pre-trained speech synthesis model based on the standard-length training data, the long-sentence training data, and the short-sentence training data to obtain a trained speech synthesis model; wherein the long-sentence training data is obtained by concatenating the standard-length training data, and the short-sentence training data is obtained by segmenting the standard-length training data.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the speech synthesis model training method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the speech synthesis model training method as described in any one of claims 1 to 10.

Images