Training method, device, electronic device and storage medium for speech synthesis model

Abstract

The present disclosure relates to a training method, device, electronic device and storage medium for a speech synthesis model. The method includes: obtaining first training data and second training data, the first training data is used to train a first preset model, and the second training data is used to train a second preset model; using the first training data to train the first preset model to obtain a first model; and using the second training data to train the second preset model to obtain a front-end model; building a target model based on the first model; building a speech synthesis model based on the target model and the front-end model; the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio. In this embodiment, the front-end model can pre-process the text to be recognized and the target model can convert the pre-processed text into the target audio, which can be applied to business scenarios that require speech to improve the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of speech synthesis technology, and in particular to a training method, device, electronic device and storage medium for a speech synthesis model. Background Art

[0002] With the popularization of intelligent devices and the development of speech recognition technology, people's interaction methods have gradually shifted from traditional text to more humanized voice interaction. Speech synthesis technology can give machines human voices, changing the traditional text interaction method. Summary of the invention

[0003] The present disclosure provides a training method, device, electronic device and storage medium for a speech synthesis model to address the deficiencies of related technologies.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for training a speech synthesis model is provided, the method comprising:

[0005] Acquire first training data and second training data, where the first training data is used to train a first preset model, and the second training data is used to train a second preset model;

[0006] Using the first training data to train the first preset model to obtain a first model; and using the second training data to train the second preset model to obtain a front-end model;

[0007] Building a target model based on the first model;

[0008] A speech synthesis model is constructed based on the target model and the front-end model; the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio.

[0009] Optionally, the first training data includes first original data and its corresponding annotation labels; the first original data includes first original text and speech data generated based on the first original text, and the annotation labels corresponding to the first original data include at least one of the following: tone labels, rhythm labels, personification labels and emotion classification labels.

[0010] Optionally, obtaining first training data includes:

[0011] Acquire a first original text and voice data corresponding to the first original text to obtain first original data;

[0012] Annotate the first original text according to the voice data corresponding to the first original text to obtain an annotation label corresponding to the first original text;

[0013] The first original data and its corresponding annotated labels are used as first training data.

[0014] Optionally, the first original text includes a first text, a second text and a third text; the first text is a text whose number of phonemes exceeds a phoneme number threshold, the second text is a modal particle text, and the third text is a text that represents an abnormal pronunciation that meets the requirements of anthropomorphic pronunciation.

[0015] Optionally, using the first training data to train the first preset model to obtain a first model includes:

[0016] Preprocessing the first training data to obtain preprocessed data;

[0017] Based on the annotated labels in the first training data and the preprocessed data, the first input data of the first preset model is obtained; and the second input data, the third input data and the fourth input data of the first preset model are obtained according to the preprocessed data; the first input data refers to the data obtained by sorting the preprocessed data and the annotated labels in a set order; the second input data refers to the data obtained by calculating the Log logarithm of the number of phoneme pronunciation frames in the preprocessed data; the third input data refers to the target alignment matrix converted from the number of phoneme pronunciation frames; the fourth input data refers to the Mel-amplitude spectrum parameters of the preprocessed data;

[0018] Inputting the first input data into the first preset model to obtain first output data, second output data, third output data and fourth output data output by the first preset model;

[0019] Obtaining a loss value of a loss function corresponding to the first preset model based on the first input data, the second input data, the third input data, the fourth input data, the first output data, the second output data, the third output data, and the fourth output data;

[0020] When the loss value exceeds a preset first loss value threshold, the parameters of the first preset model are adjusted according to a preset adjustment method until the loss value is less than the first loss value threshold and the training is stopped to obtain the first model.

[0021] Optionally, obtaining first input data of the first preset model based on the annotation labels in the first training data and the preprocessed data includes:

[0022] The pre-processed data and the marked labels are arranged in a set order to form first input data.

[0023] Optionally, obtaining second input data, third input data and fourth input data of the first preset model according to the preprocessed data includes:

[0024] Calculating the logarithm of the number of phoneme pronunciation frames in the preprocessed data to obtain second input data;

[0025] Convert the phoneme pronunciation frame number into a target alignment matrix to obtain third input data;

[0026] Acquire Mel-amplitude spectrum parameters in the preprocessed data to obtain fourth input data.

[0027] Optionally, obtaining a loss value of a loss function corresponding to the first preset model based on the first input data, the second input data, the third input data, the fourth input data, the first output data, the second output data, the third output data, and the fourth output data includes:

[0028] Calculate a phoneme classification prediction loss value according to the first input data and the fourth output data, and use the audio classification prediction loss value as a first loss value;

[0029] Calculate a phoneme duration prediction loss value according to the second input data and the second output data, and use the phoneme duration prediction loss value as a second loss value;

[0030] Calculate an alignment prediction loss value according to the third input data and the third output data, and use the alignment prediction loss value as a third loss value;

[0031] Calculating an acoustic parameter predicted loss value according to the fourth input data and the first output data, and using the acoustic parameter predicted loss value as a fourth loss value;

[0032] A loss value of the loss function is calculated based on the first loss value, the second loss value, the third loss value, the fourth loss value, and respective weight values.

[0033] Optionally, the weight values ​​of the first loss value, the second loss value, the third loss value and the fourth loss value are respectively set based on the following principle:

[0034] The product of the first loss value and its weight value, the product of the second loss value and its weight value, the product of the third loss value and its weight value, and the product of the fourth loss value and its weight value, the sizes of each product are within the same preset range;

[0035] The weight values ​​of the fourth loss value, the second loss value, the third loss value and the first loss value decrease in sequence, and a ratio of two adjacent weight values ​​exceeds a set value.

[0036] Optionally, the structure of the first preset model includes: a text encoding module, a phoneme filtering module, an addition module, a phoneme classification module, a duration prediction module, an attention mechanism module and an acoustic decoding module; the phoneme filtering module is respectively connected to the text encoding module, the addition module, the phoneme classification module and the duration prediction module; the addition module is respectively connected to the second to last hidden layer in the duration prediction module, the acoustic decoding module and the attention mechanism module;

[0037] The input data of the text encoding module is the first input data;

[0038] The input data of the attention mechanism module also includes fourth input data;

[0039] The output data of the acoustic decoding module is the predicted acoustic parameter, and the predicted acoustic parameter is used as the first output data;

[0040] The output data of the duration prediction module is the predicted phoneme duration, and the predicted phoneme duration is used as the second output data;

[0041] The output data of the attention mechanism module is a prediction alignment matrix, and the prediction alignment matrix is ​​used as the third output data;

[0042] The output data of the phoneme classification module is phoneme classification data, and the phoneme classification data is used as fourth output data.

[0043] Optionally, the target model includes a second model, and constructing the target model based on the first model includes:

[0044] Adjust the structure of the first model according to a first preset method to obtain the structure of the second model;

[0045] Migrating parameter data of the first model to the second model; the second model is used to output the target audio;

[0046] The first preset method includes: removing the phoneme classification module in the first model; retaining the first output in the first model; and using the output data of the acoustic decoding module in the first model as input data of the attention mechanism module in the first model.

[0047] Optionally, the target model includes a third model, and constructing the target model based on the first model includes:

[0048] Adjust the structure of the first model according to a second preset method to obtain the structure of a third model;

[0049] Migrating the parameter data of the first model to the third model; the second model is used to output the target audio;

[0050] The second preset method includes: removing the phoneme classification module in the first model; retaining the first output in the first model; replacing the attention mechanism module in the first model with a phoneme expansion module; and using the output data of the acoustic decoding module in the first model as input data of the phoneme expansion module.

[0051] Optionally, the second training data includes second original data and corresponding label data; the second original data includes second original text and speech data generated according to the second original text, and the label data corresponding to the second original data includes at least one of the following: a classification label, a type label, a speech speed label, and an abnormal pronunciation label;

[0052] The classification label includes at least one of the following: an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label, a sentence type classification label or an emotion type classification label; the type label includes at least one of the following: a connected reading type label, a repetition type label and a reverse reading type label; the abnormal pronunciation label includes at least one of the following: a dragged sound label, a swallowed sound label and a lazy sound label.

[0053] Optionally, obtaining second training data includes:

[0054] Acquire a second original text and voice data corresponding to the second original text to obtain second original data;

[0055] The second original data is processed to obtain label data corresponding to the second original data; and the second original data and the label data are used as the second training data.

[0056] Optionally, processing the second original data to obtain label data corresponding to the second original data includes:

[0057] The second original text is classified by a preset classification method to obtain a classification label; the classification label includes at least one of the following: an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label, a sentence type classification label or an emotion type classification label;

[0058] or,

[0059] The second original text is analyzed by using a context feature analysis method to obtain a type label; the type label includes at least one of the following: a continuous reading type label, a repeated type label, and a reverse reading type label;

[0060] or,

[0061] The second original text and its corresponding speech data are aligned by a forced alignment method to obtain the pronunciation duration of each syllable; a speech speed label is generated based on the pronunciation duration of each syllable, and an abnormal pronunciation label is generated based on the pronunciation duration and energy parameter of each syllable; the abnormal pronunciation label includes at least one of the following: a dragging sound label, a swallowing sound label, and a lazy sound label;

[0062] The classification label, the type label, the speech speed label and the abnormal pronunciation label are used as label data corresponding to the second original data.

[0063] Optionally, using the second training data to train the second preset model to obtain a front-end model includes:

[0064] Inputting the second original text in the second training data into a second preset model, and obtaining predicted label data output by the second preset model;

[0065] Obtaining a loss value of a loss function corresponding to the second preset model based on the label data corresponding to the second original text and the predicted label data;

[0066] When the loss value exceeds a preset second loss value threshold, the parameters of the second preset model are adjusted according to a preset adjustment method until the loss value is less than the second loss value threshold and the training is stopped to obtain the front-end model.

[0067] According to a second aspect of an embodiment of the present disclosure, a speech synthesis method is provided, the method comprising:

[0068] Get the text to be synthesized;

[0069] Inputting the text to be synthesized into a preset speech synthesis model for speech synthesis, and outputting target audio;

[0070] The speech synthesis model is constructed based on the target model and the front-end model. The target model is constructed based on a first model obtained by training a first preset model using first training data; the front-end model is a front-end model obtained by training a second preset model using second training data.

[0071] Optionally, the first training data includes first original data and its corresponding annotation labels; the first original data includes first original text and speech data generated based on the first original text, and the annotation labels corresponding to the first original data include at least one of the following: tone labels, rhythm labels, personification labels and emotion classification labels.

[0072] Optionally, the first original text includes a first text, a second text and a third text; the first text is a text in which the number of phonemes exceeds a phoneme number threshold, the second text is a modal particle text, and the third text is a text that characterizes and meets the requirements of anthropomorphic abnormal pronunciation.

[0073] Optionally, the structure of the first preset model includes: a text encoding module, a phoneme filtering module, an addition module, a phoneme classification module, a duration prediction module, an attention mechanism module and an acoustic decoding module; the phoneme filtering module is respectively connected to the text encoding module, the addition module, the phoneme classification module and the duration prediction module; the addition module is respectively connected to the second to last hidden layer in the duration prediction module, the acoustic decoding module and the attention mechanism module;

[0074] The input data of the text encoding module is the first input data; the first input data refers to the data obtained by sorting the preprocessed data and marking the labels in a set order;

[0075] The input data of the attention mechanism module also includes fourth input data; the fourth input data refers to the Mel magnitude spectrum parameter of the preprocessed data;

[0076] The output data of the acoustic decoding module is the predicted acoustic parameter, and the predicted acoustic parameter is used as the first output data;

[0077] The output data of the duration prediction module is the predicted phoneme duration, and the predicted phoneme duration is used as the second output data;

[0078] The output data of the attention mechanism module is a prediction alignment matrix, and the prediction alignment matrix is ​​used as the third output data;

[0079] The output data of the phoneme classification module is phoneme classification data, and the phoneme classification data is used as fourth output data.

[0080] Optionally, the second training data includes second original data and corresponding label data; the second original data includes second original text and speech data generated according to the second original text, and the label data corresponding to the second original data includes at least one of the following: a classification label, a type label, a speech speed label, and an abnormal pronunciation label;

[0081] The classification label includes at least one of the following: an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label, a sentence type classification label or an emotion type classification label; the type label includes at least one of the following: a connected reading type label, a repetition type label and a reverse reading type label; the abnormal pronunciation label includes at least one of the following: a dragged sound label, a swallowed sound label and a lazy sound label.

[0082] According to a third aspect of an embodiment of the present disclosure, a training device for a speech synthesis model is provided, the device comprising:

[0083] A training data acquisition module is configured to acquire first training data and second training data, wherein the first training data is used to train a first preset model, and the second training data is used to train a second preset model;

[0084] A first model acquisition module is configured to train the first preset model using the first training data to obtain a first model;

[0085] A front-end model acquisition module, configured to train the second preset model using the second training data to obtain a front-end model;

[0086] A target model building module, configured to build a target model based on the first model;

[0087] The speech module construction module is configured to construct a speech synthesis model based on the target model and the front-end model; the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio.

[0088] According to a fourth aspect of an embodiment of the present disclosure, a speech synthesis device is provided, the device comprising:

[0089] A text acquisition module, configured to acquire the text to be synthesized;

[0090] An audio acquisition module is configured to input the text to be synthesized into a preset speech synthesis model for speech synthesis and output a target audio;

[0091] The speech synthesis model is constructed based on a target model and a front-end model. The target model is constructed based on a first model obtained by training a first preset model using first training data; the front-end model is a front-end model obtained by training a second preset model using second training data.

[0092] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0093] processor;

[0094] a memory for storing a computer program executable by the processor;

[0095] The processor is configured to execute the computer program in the memory to implement the above method.

[0096] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, the method as described above can be implemented.

[0097] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0098] It can be seen from the above embodiments that the scheme provided by the embodiments of the present disclosure can obtain the first training data and the second training data, the first training data is used to train the first preset model, and the second training data is used to train the second preset model; then, the first preset model is trained using the first training data to obtain the first model; and the second preset model is trained using the second training data to obtain the front-end model; then, a target model is constructed based on the first model; finally, a speech synthesis model is constructed based on the target model and the front-end model; the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio. In this way, the front-end model in this embodiment can pre-process the text to be recognized and the target model can convert the pre-processed text into the target audio, which can be applied to business scenarios that require speech and improve the user experience.

[0099] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0101] Figure 1 The present invention is a flowchart of a method for training a speech synthesis model according to an exemplary embodiment.

[0102] Figure 2 The figure is a flowchart of obtaining a label according to an exemplary embodiment.

[0103] Figure 3 The figure is a schematic diagram showing the effect of labeling according to an exemplary embodiment.

[0104] Figure 4 The figure is a flowchart of obtaining second training data according to an exemplary embodiment.

[0105] Figure 5 The figure is a flowchart of predicting labels according to an exemplary embodiment.

[0106] Figure 6 The figure is a flowchart of obtaining a front-end model according to an exemplary embodiment.

[0107] Figure 7 The figure is a flowchart of a training front-end model according to an exemplary embodiment.

[0108] Figure 8 The present invention is a flowchart of a front-end model predicting a label according to an exemplary embodiment.

[0109] Fig. 9 The figure is a flowchart of training a first model according to an exemplary embodiment.

[0110] Fig.10 It is a structural block diagram of a first model according to an exemplary embodiment.

[0111] Fig.11 The figure is a flowchart of calculating the loss value of the first preset model according to an exemplary embodiment.

[0112] Fig.12 is a structural block diagram of a second model according to an exemplary embodiment.

[0113] Fig.13 is a structural block diagram of a third model according to an exemplary embodiment.

[0114] Fig.14 It is a structural block diagram of a speech synthesis model according to an exemplary embodiment.

[0115] Fig.15 It is a workflow diagram of a speech synthesis model according to an exemplary embodiment.

[0116] Fig.16 The figure is a flow chart of a speech synthesis method according to an exemplary embodiment.

[0117] Fig.17 It is a block diagram of a training device for a speech synthesis model according to an exemplary embodiment.

[0118] Fig.18 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0119] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described exemplarily below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the attached claims. It should be noted that the features in the following embodiments and implementations may be combined with each other without conflict.

[0120] In order to solve the above technical problems, the embodiments of the present disclosure provide a method for training a speech synthesis model, which can be applied to electronic devices with a speech playback function. Figure 1 is a flowchart of a method for training a speech synthesis model according to an exemplary embodiment. Figure 1 , a training method for a speech synthesis model, comprising steps 11 to 14.

[0121] In step 11, first training data and second training data are obtained, wherein the first training data is used to train a first preset model, and the second training data is used to train a second preset model.

[0122] In this embodiment, the electronic device can obtain the first training data. Figure 2 , including step 21 to step 22. In step 21, the electronic device can obtain the first original text and the voice data corresponding to the first original text (hereinafter also referred to as the first original voice data) to obtain the first original data. Considering that the target audio output by the speech synthesis model of the present invention has anthropomorphic characteristics, a super anthropomorphic recording scheme is adopted in this step. For example, the recording personnel uses the recording equipment in a professional recording studio to record according to the first original text to obtain the voice data corresponding to the first original text, and the first original text and its corresponding voice data are formed into a set of data pairs and stored in a specified location, such as a local storage device or a cloud.

[0123] It should be noted that in this step, the first original text is anthropomorphized and includes: a first text, a second text, and a third text. The first text is a text whose number of phonemes exceeds the threshold of the number of phonemes, such as ordinary texts such as articles, papers, news, etc. in actual applications; the above-mentioned threshold of the number of phonemes is set according to the specific scenario. Taking the first text as a Chinese and English text as an example, the threshold of the number of phonemes can be set to 147. In other words, the phonemes in the first text need to cover 147 phonemes. The second text is an interjection text, including interjections such as ah, oh, and uh; the third text is a text that represents abnormal pronunciations that meet the requirements of anthropomorphism, including abnormal pronunciations such as stuttering, dragging, and sighing.

[0124] It should be noted that the proportions of the first text, the second text and the third text in the first original text are the first proportion, the second proportion and the third proportion respectively. For example, the first proportion ranges from 45% to 55%, the second proportion ranges from 15% to 25%, and the third proportion ranges from 25% to 35%, and the sum of the first proportion, the second proportion and the third proportion is 1. And the setting principles of the first proportion, the second proportion and the third proportion respectively include: (1) The first proportion needs to meet the requirement of balanced phoneme coverage, which can be understood as the first model trained with only the first text can output the target audio that can be pronounced normally, meeting the minimum requirement of speech synthesis. (2) The second proportion needs to meet the tone of anthropomorphic pronunciation, so as to achieve an anthropomorphic effect on the basis of normal pronunciation. (3) The third proportion needs to meet the personalized needs of different users when pronouncing, so as to achieve a further anthropomorphic (or super anthropomorphic) effect on the basis of the anthropomorphic effect, so as to achieve the effect of consistency between human and machine pronunciation. Based on the above principles, in one example, the first ratio is 50%, the second ratio is 20, and the third ratio is 30%. This can reproduce the light reading, erhua, swallowing, lazy sound, bubbling sound, adding filler words, and repeating natural phenomena in spoken language, which is conducive to improving the accuracy of anthropomorphic speech synthesis in subsequent processes.

[0125] In step 22, the electronic device may annotate the first original text according to the voice data corresponding to the first original text to obtain an annotation label corresponding to the first original text. The above annotation label may be stored separately or directly inserted into the morpheme sequence corresponding to the first original text. In one example, the annotation process is as follows:

[0126] 1) Pinyin with tones, the pinyin annotation is consistent with the content of the first original voice data, the effect is as follows Figure 3 See Figure 3 In the annotated content zhu4 nin3#3sheng1 ri4 kuai4 le4#3, the tone of zhu is the fourth tone, so the label 4 is added after zhu; the tone of nin is the third tone, so the label 3 is added after nin, and so on.

[0127] 2) Prosodic pause information, including prosodic word boundaries, prosodic phrase boundaries, and intonation phrase boundaries. Figure 3 See Figure 3 In the annotated content zhu4 nin3#3sheng1 ri4 kuai4 le4#3, the rhythmic pause between nin and sheng is longer, so the rhythmic label #3 is marked, where the number 3 represents the length of the pause, and the longer the time, the larger the number.

[0128] 3) Anthropomorphic labels refer to labels that represent the emotional and pronunciation characteristics of human speech, including dragging sound <%DRAG%>, stuttering <%RP-SYL%>, backward reading <%RP-WRD%>, swallowing sound <%SWALLOW%>, lazy sound <%LAZY%>, breathing sound <#BREATH#>, laughing <#LAUFHT#>, sighing <#SIGH#>, crying <#CRY#>, humming (angry) <#HNG-ANGRY#>, humming (sarcasm) <#HNG-TAUNT#>, lip smacking <#SMACK#>, speaking speed <$0><$1><$2>, etc., among which:

[0129] 3.1) Dragging sound: an anthropomorphic tag used to reflect the emotional effects of dragging sound and hesitation. The reference format is: <%DRAG%>;

[0130] 3.2) Stuttering: An anthropomorphic label used to reflect the emotional effects of hesitating and nervousness. The reference format is: <%RP-SYL%>;

[0131] 3.3) Reverse reading: an anthropomorphic label used to reflect the emotional effect of repeated emphasis, reference format: <%RP-WRD%>;

[0132] 3.4) Breathing sound: an anthropomorphic tag used to increase the effect of human pronunciation characteristics, reference format: <#BREATH#>;

[0133] 3.5) Laughter: An anthropomorphic label used to increase the effect of artificial pronunciation characteristics, different from the traditional "hahaha" reading method, reference format: <#LAUFHT#>;

[0134] 3.6) Sigh: An anthropomorphic label used to increase the effect of human pronunciation characteristics, different from the traditional "sigh" reading method, reference format: <#SIGH#>;

[0135] 3.7) Crying: An anthropomorphic label used to increase the effect of artificial pronunciation characteristics, which is different from the traditional "woohoo" pronunciation method. The reference format is: <#CRY#>;

[0136] 3.8) Hum: An anthropomorphic label used to increase the effect of human pronunciation characteristics, different from the traditional "hum" reading method. The reference format is: <#HNG-ANGRY#> for anger, <#HNG-TAUNT#> for sarcasm;

[0137] 3.9) Swallowing sound: an anthropomorphic label used to increase the effect of human pronunciation characteristics, reference format: <% SWALLOW%>;

[0138] 3.10) Lazy sound: an anthropomorphic tag used to increase the effect of artificial pronunciation characteristics, reference format: <%LAZY%>;

[0139] 3.11) Speech speed: an anthropomorphic label used to reflect different emotional states through speech speed. The reference format is: <$0><$1><$2> represents fast, normal, and slow respectively;

[0140] 3.12) Lip smacking sound: an anthropomorphic tag used to increase the effect of artificial pronunciation characteristics, reference format: <#SMACK#>;

[0141] 4) Overall features of sentences, through continuous emotion embedding coding to control the emotion category of the synthesized emotion control sentence, including sentence type labels or emotion classification labels such as T0, T1, T2, E0, E1, E2, which respectively represent declarative sentences, exclamatory sentences, interrogative sentences, neutral emotions, happy emotions, and sad emotions. Among them, the emotion classification label refers to the label that represents the overall emotion of the sentence. The above emotion classification labels are helpful to help the subsequent text to be synthesized and the context predict the emotion coding to control the tone of the sentence, and further enhance the effect of anthropomorphism of the audio sentence.

[0142] It should be noted that the above labeling process can also be implemented manually, which is not limited here.

[0143] Therefore, based on the above description, the electronic device can obtain the annotation label corresponding to the first original text, including at least one of the following: tone label, rhythm label, personification label and emotion classification label. It should be noted that the annotation label corresponding to the first original text can be set according to the specific scenario. The above content is only used to illustrate the setting format of various labels and the similarities and differences between different labels. If different labels and their functions can be identified, the corresponding solution falls within the protection scope of this disclosure.

[0144] In this way, the electronic device can use the first original data and its corresponding annotation label as the first training data, thereby obtaining training data for training the first preset model.

[0145] In this embodiment, the electronic device can obtain the second training data. Figure 4 , including steps 41 to 42.

[0146] In step 41, the electronic device can obtain the second original text and the voice data corresponding to the second original text to obtain the second original data. Considering that the amount and complexity of the training data required for the second preset model are large, this step can be implemented using data in an open source database, and the training data can be generated using a statistical analysis method. If a (text-speech) database recorded by multiple people in different environments is used, the amount of the second original data exceeds 100,000 sentences.

[0147] In step 42, the electronic device may process the second original data to obtain label data corresponding to the second original data.

[0148] See also Figure 5 , the electronic device can classify the second original text using a preset classification method (such as FastText classifier, TextCNN classifier, etc., which can be adjusted) to obtain a classification label. For example, according to the Chinese characters and the corresponding pinyin sequence, generate an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label; according to the sentence and the corresponding punctuation mark, generate a sentence type or emotion type classification label, etc. That is, the above classification label includes at least one of the following: an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label, a sentence type classification label, or an emotion type classification label.

[0149] Continue to see Figure 5 , the electronic device can analyze the second original text using a context feature analysis method to obtain a type label. For example, a regular matching method is used to determine whether there are types such as connected reading, repetition, and reverse reading, and a corresponding label is generated; the above type label includes at least one of the following: a connected reading type label, a repeated type label, and a reverse reading type label. In practical applications, the frequency threshold of the types such as connected reading and repetition can also be determined based on the statistical analysis of the repetition rate, and the above frequency threshold can be used as a reference threshold for repeated text filling and redundant text deletion.

[0150] Continuing to refer to, the electronic device can adopt a forced alignment method (a forced alignment tool in the related technology) to align the second original text and its corresponding voice data to obtain the pronunciation duration of each syllable; and generate a speech speed label based on the pronunciation duration of each syllable, and generate an abnormal pronunciation label based on the pronunciation duration and energy parameters of each syllable; the above-mentioned abnormal pronunciation label includes at least one of the following: a dragging sound label, a swallowing sound label and a lazy sound label.

[0151] In this way, the electronic device can use the above-mentioned classification label, the above-mentioned type label, the above-mentioned speech speed label and the above-mentioned abnormal pronunciation label as label data corresponding to the second original data, and use the above-mentioned second original data and the above-mentioned label data as the second training data for training the second preset model.

[0152] In step 12, the first preset model is trained using the first training data to obtain a first model; and the second preset model is trained using the second training data to obtain a front-end model.

[0153] In this embodiment, the electronic device can use the second training data to train the second preset model to obtain a front-end model. Figure 6 , including steps 61 to 63.

[0154] In step 61, the electronic device may input the second original text in the second training data into the second preset model, and obtain the preset label data output by the second preset model. The above-mentioned second preset model may be implemented using a bidirectional long short-term memory model (BLSTM). The structure of the BLSTM may refer to the relevant technology, which will not be described in detail here.

[0155] Taking the training of BLSTM model as an example, the training structure is as follows Figure 7 See Figure 7 , the electronic device can convert the second original text into vectorized text and input it into the BLSTM model. Then, the electronic device can obtain the predicted label data output by the BLSTM model.

[0156] It should be noted that Figure 7 In the example, a fully connected layer (FC layer) is added before and after the BLSTM model. The first FC layer converts the input data into data of a specified dimension, such as 512-dimensional input data. The second FC layer converts the output data of BLSTM into output dimensions, thereby ensuring that the dimensions of the input and output data can meet the requirements of the modules connected before and after.

[0157] In step 62, the electronic device may obtain the loss value of the loss function corresponding to the second preset model based on the label data corresponding to the second original text and the predicted label data. After obtaining the predicted data output by the second preset model, the electronic device may use the difference between the statistical predicted label data and the label data corresponding to the second original text (i.e., softmax cross entropy) as the loss value of the loss function.

[0158] In step 63, when the loss value exceeds the preset second loss value threshold, the electronic device can adjust the parameters of the second preset model according to a preset adjustment method (such as a gradient reverse adjustment method), and continue to perform steps 61 and 62 until the loss value is less than the second loss value threshold, and stop training to obtain a front-end model. The above front-end model has the effect of labeling the input text.

[0159] It should be noted that in this example, the front-end model includes a text addition and deletion model (also referred to as the fourth model), an interjection pronunciation model (also referred to as the fifth model), and an anthropomorphic label prediction model (also referred to as the sixth model). Figure 8 ,The working process of the front-end model may include: any text is converted into vectorized text and then input into the front-end model, and the front-end model can predict the corresponding label data.

[0160] For example

[0161] The fourth model: the input is vectorized text, such as "this is too exaggerated", and the output is the number of repetitions of the corresponding Chinese characters, such as: [4 1 1 1 1 1 1], which means "this is too exaggerated";

[0162] The fifth model: The input is vectorized text, such as "Ah, you missed the bus again", "Ah, I won't dare to do it next time", and the output is the pronunciation type of modal particles, such as: [0 0 1 0 0 0], [0 0 0 0 1 0], which respectively represent "Ah (a2), you missed the bus again", "Ah (a4), I won't dare to do it next time";

[0163] The sixth model: The input is vectorized text, such as: "Sometimes I feel lonely", and the output is a personified feature label matrix of a specified dimension (such as 12 dimensions), such as: the first dimension "drag" [0 0 1 0 0 1 0 0 0 0], which means "Sometimes <%DRAG%> I also <%DRAG%> feel lonely".

[0164] In this way, in this embodiment, the second preset model is trained by the second training data to obtain the final front-end data, and the front-end model can be used to annotate the input text to be synthesized in the subsequent speech synthesis model.

[0165] In this embodiment, the electronic device can use the first training data to train the first preset model to obtain the first model. Fig. 9 , including steps 91 to 95.

[0166] In step 91, the electronic device may preprocess the first training data to obtain preprocessed data. The above preprocessing may include four parts: text normalization (TN), word segmentation part of speech prediction, prosody level prediction, and pronunciation prediction (G2P). Among them, text normalization (TN) is also called text cleaning or conversion, including steps such as text cleaning, case conversion, word correction, stop word deletion, stem extraction, and word type restoration. It is to standardize text data into a format that can be used by NLP, analysis systems, and applications.

[0167] Word segmentation part-of-speech prediction refers to segmenting the delimiters between different words in the first original text and obtaining the part-of-speech of each word. The above part-of-speech can be used to describe the role of a word in the context, and part-of-speech tagging is to identify the part-of-speech of these words to determine their role in the context.

[0168] The prosodic level mainly includes: prosodic words, prosodic phrases and intonation phrases. Prosodic words are relatively stable and have a high prediction accuracy; prosodic phrases are relatively flexible; intonation phrases can be regarded as a group of prosodic phrases, which often produce obvious pauses. They are a larger hierarchical structure and are more critical to people's hearing and understanding. Prosodic level prediction is to obtain the above-mentioned prosodic level, obtain the pauses in the first original text based on the speech data corresponding to the first original text, and use models such as conditional random fields (CRF) and recurrent neural networks (RNN) to predict the prosodic level.

[0169] Pronunciation prediction (G2P) can use RNN model and LSTM model to realize the conversion from English words to phonemes, so as to obtain the pronunciation duration of each phoneme in the speech data.

[0170] It should be noted that the above preprocessing process can be adjusted according to the actual scenario. When the preprocessing data of the first original text can be obtained, the corresponding solution falls within the protection scope of the present disclosure.

[0171] In step 92, the electronic device may obtain first input data of the first preset model based on the annotation labels in the first training data and the preprocessed data; and obtain second input data, third input data and fourth input data of the first preset model based on the preprocessed data.

[0172] In this embodiment, the electronic device may arrange the preprocessed data and the annotated labels in a set order to form the first input data, that is, the first input data refers to the data obtained by sorting the preprocessed data and the annotated labels in the set order. For example, the electronic device may arrange the phoneme sequence, tone label, pause label and the annotated label in the preprocessed data in sequence to obtain the first input data.

[0173] In this embodiment, the electronic device can calculate the Log logarithm of the number of phoneme pronunciation frames in the preprocessed data to obtain the second input data, that is, the second input data refers to the data obtained by calculating the Log logarithm of the number of phoneme pronunciation frames in the preprocessed data. The above-mentioned number of phoneme pronunciation frames can be set to a frame shift (Hop size) of H while extracting the Mel amplitude spectrum parameter, i.e., the energy parameter, and the phoneme pronunciation duration (in millimeters) is divided by the above-mentioned frame shift H to obtain the number of phoneme pronunciation frames.

[0174] In this embodiment, the electronic device can convert the number of frames of phoneme pronunciation into a target alignment matrix to obtain the third input data, that is, the third input data refers to the target alignment matrix converted from the number of frames of phoneme pronunciation. For example, given the number of frames of N phonemes, which are d1, d2, …, dN respectively, convert them into a target alignment matrix W, corresponding to a matrix of size N*M, where M = d1 + d2 + … + dN; the conversion method is: when i and j satisfy the following conditions, set the element Wij corresponding to the i-th row and j-th column of the matrix W to 1, otherwise set it to 0. Among them, the conditions are: for each i, i belongs to [1, N], and Si ≤ j < Ei. Where Si represents the start position of the i-th phoneme Si = d1 + d2 + … + di-1, and Ei represents the end position of the i-th phoneme Ei = Si + di.

[0175] In this embodiment, the electronic device can obtain the Mel-scale cepstral coefficients (MFCC) in the preprocessed data to obtain the fourth input data, that is, the fourth input data refers to the Mel-scale cepstral coefficients of the preprocessed data. For example, the electronic device can perform pre-emphasis, framing, and windowing on the speech data in the first training data; then, for each short-time analysis window, obtain the corresponding spectrum through FFT; after that, obtain the Mel spectrum through the Mel filter bank for the above spectrum; finally, perform cepstral analysis on the Mel spectrum (take the logarithm, perform the inverse transform, and the actual inverse transform is generally implemented through the discrete cosine transform DCT, and take the 2nd to 13th coefficients after DCT as the MFCC coefficients) to obtain the Mel-frequency cepstral coefficients MFCC.

[0176] It should be noted that the above second input data, third input data, and fourth input data are used to calculate the loss value (loss) of the loss function of the first preset model and are not used temporarily when training the first preset model.

[0177] In step 93, the electronic device can input the first input data into the first preset model to obtain the first output data, second output data, third output data, and fourth output data output by the first preset model.

[0178] In this embodiment, the first preset model can be pre-stored in the electronic device. Refer to Fig.10 , the structure of the first preset model includes: a text encoding module, a phoneme filtering module, an addition module, a phoneme classification module, a duration prediction module, an attention mechanism module, and an acoustic decoding module. The phoneme filtering module is respectively connected to the text encoding module, the addition module, the phoneme classification module, and the duration prediction module; the addition module is respectively connected to the penultimate hidden layer in the duration prediction module, the acoustic decoding module, and the attention mechanism module. Among them,

[0179] The input data of the text encoding module is the first input data. In addition to the output data of the addition module, the input data of the attention mechanism module also includes the fourth input data. The output data of the acoustic decoding module is the predicted acoustic parameters, and the predicted acoustic parameters are used as the first output data; the output data of the duration prediction module is the predicted phoneme duration, and the predicted phoneme duration is used as the second output data, so that the start time and end time of each phoneme can be obtained, that is, the start and end timestamps of each phoneme can be accurately output, which improves the accuracy of speech speed control and is conducive to controlling and aligning the lip shape when playing the target audio later.

[0180] The output data of the attention mechanism module is the predicted alignment matrix, and the predicted alignment matrix is ​​used as the third output data. The output data of the phoneme classification module is the phoneme classification data, and the phoneme classification data is used as the fourth output data.

[0181] It should be noted that each module in the first preset model is implemented using a corresponding network model, and the specific implementation can be implemented using a solution in the relevant technology. If the corresponding function can be achieved, the corresponding solution falls within the protection scope of this disclosure.

[0182] It should be noted that Fig.10 In the first preset model shown, solid lines and dotted lines are used to represent connecting lines and frames, respectively. Among them, the solid frame and solid connecting lines indicate that this module exists in the model training and use stage, and the dotted frame and dotted connecting lines indicate that this module is only used in the model training stage, or the dotted frame and dotted connecting lines are removed in the use stage.

[0183] In this embodiment, the electronic device can input the first input data into the first preset model. The output data of the text encoding module in the first preset model is implicit state data, such as 100 256-dimensional vectors, which represent the number of phonemes and the state data occupied by each phoneme, so that each input code contains context information, thereby making the pronunciation clearer. For example, the above-mentioned text encoding module can be implemented using a Tacotron model (e.g., a 3-layer convolution + 1-layer LSTM structure), and its output is an encoding result.

[0184] The phoneme filtering module can process the above-mentioned implicit state data, that is, remove some columns in the implicit state data, such as removing the column corresponding to the prosody label (#3), so that each vector corresponds to a pronunciation duration.

[0185] The data output by the phoneme filtering module can be input into the phoneme classification module and the duration prediction module respectively. The phoneme classification module can predict the phoneme classification of the implicit state data to obtain the fourth output data. The duration prediction module can predict the pronunciation duration of the implicit state data, and use the predicted phoneme duration as the second output data. At the same time, the output data of the duration prediction module and the output data of the phoneme filtering module are obtained after passing through the addition module, and the above-mentioned summed data is input into the acoustic decoding module and the attention mechanism module.

[0186] It should be noted that there are two ways to implement the addition module: the first is to quantize the duration into an integer, convert it into an embedded representation of the duration through an embedded lookup table, and then add it to the phoneme filter module; the second is to add the output of the second-to-last hidden layer of the duration prediction module to the phoneme filter module.

[0187] After the attention mechanism module obtains the above-mentioned summed data and the fourth input data, it can obtain the predicted alignment matrix, that is, obtain the third output data. And the above-mentioned predicted alignment matrix is ​​input into the acoustic decoding module. Since the encoding information of the phoneme duration is added to the attention mechanism module, the attention mechanism module can determine the start and end positions of the phoneme pronunciation, so that when predicting the current frame, it can know in advance which phoneme should be noticed, and how long it will take to stay on this phoneme in the future. In other words, the addition of phoneme duration encoding makes the training more stable, which is conducive to improving the convergence speed of the training process and improving the training efficiency. In the subsequent use process, the increase in the number of phoneme frames can help control the pronunciation duration of each phoneme during synthesis to avoid the situation where the pronunciation is too short or too long.

[0188] The acoustic decoding module may process the above-mentioned summed data and the predicted alignment matrix to obtain predicted acoustic parameters, that is, to obtain first output data.

[0189] In step 94, the electronic device may obtain the loss value of the loss function corresponding to the first preset model based on the first input data, the second input data, the third input data, the fourth input data, the first output data, the second output data, the third output data and the fourth output data. In this step, the loss function of the first preset model is composed of four parts of loss, namely, phoneme classification loss (also referred to as the first loss value), duration prediction loss (also referred to as the second loss value), alignment prediction loss (also referred to as the third loss value), and acoustic parameter prediction loss (also referred to as the fourth loss value).

[0190] The electronic device may calculate the phoneme classification prediction loss value according to the first input data and the fourth output data, and use the audio classification prediction loss value as the first loss value. Fig.11, the electronic device can obtain the cross entropy between the fourth output data and the first input data, that is, the cross entropy between the output data of the phoneme classification module and the input data of the text encoding module.

[0191] The electronic device may calculate the phoneme duration prediction loss value according to the second input data and the second output data, and use the phoneme duration prediction loss value as the second loss value. Fig.11 The electronic device can obtain the mean square error of the second output data and the second input data after taking the natural logarithm, that is, the mean square error of the predicted phoneme duration and the actual phoneme duration after taking the natural logarithm.

[0192] The electronic device may calculate an alignment prediction loss value according to the third input data and the third output data, and use the alignment prediction loss value as the third loss value. Fig.11 , the electronic device can obtain the mean square error after taking the natural logarithm of the third output data and the third input data, that is, the mean square error between the target alignment matrix and the predicted alignment matrix (obtained after aligning the first original training data using the forced alignment tool). By increasing the alignment prediction loss value, the alignment matrix preset by the target model can be matched with the actual one, so that the pronunciation of each word in the text is more complete, and the problems of missing words, long silence at the end, or redundant pronunciation can be completely avoided, thereby improving the naturalness of the subsequent generation of the target audio.

[0193] The electronic device may calculate the acoustic parameter predicted loss value according to the fourth input data and the first output data, and use the acoustic parameter predicted loss value as the fourth loss value. Fig.11 , the electronic device can obtain the mean square error between the first output data and the fourth input data, that is, the mean square error between the real acoustic parameters and the predicted acoustic parameters (directly obtained according to the first original training).

[0194] The electronic device may calculate the loss value of the loss function based on the first loss value, the second loss value, the third loss value, the fourth loss value, and their respective weight values, as shown in the following formula:

[0195] Loss=Loss1*w1+Loss2*w2+Loss3*w3+Loss4*w4; (1)

[0196] In formula (1), Loss1, Loss2, Loss3 and Loss4 represent the first loss value, the second loss value, the third loss value and the fourth loss value respectively; w1, w2, w3 and w4 represent the weight values ​​of the first loss value, the second loss value, the third loss value and the fourth loss value respectively.

[0197] It should be noted that the weight values ​​w1, w2, w3 and w4 can be set based on the following principles:

[0198] (1) The product of the first loss value and its weight value, the product of the second loss value and its weight value, the product of the third loss value and its weight value, and the product of the fourth loss value and its weight value, the sizes of each product are within the same preset range, for example, all between [0,10] and [10,99].

[0199] (2) The weight values ​​of the fourth loss value, the second loss value, the third loss value and the first loss value decrease in sequence, and the ratio of two adjacent weight values ​​exceeds the set value, which can be 2, that is, the ratio of w1 to w2 exceeds 2. It can also be understood that the important loss values ​​are given the weight values ​​of the trap, so that the first preset model can learn which parameters are more important, so as to avoid over-optimization on unimportant parameters, which is conducive to improving training efficiency and ensuring the accuracy of the subsequent output results of the first model.

[0200] In step 95, when the loss value exceeds the preset first loss value threshold, the electronic device can adjust the parameters of the first preset model according to the preset adjustment method, until the loss value is less than the first loss value threshold, the training is stopped, and the first model is obtained. For example, when the loss value in formula (1) exceeds the first loss value threshold, the electronic device can adjust the parameters of each module in the first preset model according to the gradient back propagation method, and continue to execute steps 91 to 94, and stop training when the loss value is less than the first loss value threshold, and obtain the first model.

[0201] In step 13, a target model is constructed based on the first model.

[0202] In this embodiment, the target model may include a second model and a third model.

[0203] See also Fig.12 , the electronic device constructs the second model based on the first model, including:

[0204] The structure of the first model is adjusted according to a first preset method to obtain the structure of the second model; wherein the first preset method includes: removing the phoneme classification module in the first model; retaining the first output in the first model and removing the third output; using the output data of the acoustic decoding module in the first model as the input data of the attention mechanism module in the first model. In addition, the parameter data of the first model is migrated to the second model.

[0205] It should be noted that in the second model, the acoustic decoding module uses the real acoustic features as input at each time step, and then passes through two fully connected layers before being sent to the attention mechanism module. Among them, the right half of the second model (on the right side of the refueling module) is a loop, and each loop is a time step; the input data of time step t is the output of time step t-1. In other words, at time step t, the input data of the attention mechanism module consists of the output data of the audio decoding module at time step t-1 and the output data of the addition module at time step t.

[0206] It should be noted that the target audio output by the second model is suitable for scenarios with high requirements on voice quality, such as reading news, novels, and books in high-fidelity speakers and other devices.

[0207] See also Fig.13 , the electronic device constructs a third model based on the first model, including:

[0208] The structure of the first model is adjusted according to the second preset method to obtain the structure of the third model; wherein the second preset method includes: removing the phoneme classification module in the first model; retaining the first output in the first model and removing the third output; replacing the attention mechanism module in the first model with a phoneme expansion module; using the output data of the acoustic decoding module in the first model as the input data of the phoneme expansion module. The parameter data of the first model is migrated to the third model. Among them, the phoneme expansion module can extend the length of the output data of the acoustic decoding module by copying, including: copying each column of the output matrix of the acoustic decoding module by a certain number of copies, and the specific number of copies depends on the duration of the current phoneme prediction. In this way, the phoneme expansion module replaces the attention calculation process in the attention module by copying, and can be used as an approximate algorithm for attention calculation. The model is smaller in size and faster in operation, which is convenient for use on mobile devices such as mobile phones.

[0209] It should be noted that the target audio output by the third model is suitable for scenarios with high requirements on voice speed, such as business scenarios such as reading text messages on devices such as smartphones.

[0210] In step 14, a speech synthesis model is constructed based on the target model and the front-end model; the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio.

[0211] In this embodiment, the electronic device can construct a speech synthesis model based on the target model and the front-end model. Fig.14 , that is, the electronic device connects the fourth model, the fifth model, the sixth model and the second model (or the third model) in sequence to obtain the above-mentioned speech synthesis model. The input data of the speech synthesis model is any text, and the output data is the anthropomorphic speech data, that is, the target audio.

[0212] The working process of the above speech synthesis model is as follows: Fig.15 As shown, including:

[0213] The electronic device can input the obtained text to be synthesized into the above-mentioned speech synthesis model.

[0214] The fourth, fifth and sixth models in the speech synthesis model perform annotation processing on the above text to be synthesized in turn, i.e., predicting the addition and deletion of characters, predicting the pronunciation of modal particles, predicting anthropomorphic labels, etc., to obtain mixed feature text. For the specific process, please refer to Figure 8 The content of the examples will not be repeated here.

[0215] The electronic device can preprocess the text to be synthesized to obtain preprocessed data. Fig. 9 The content of the examples will not be repeated here.

[0216] The electronic device can annotate the above text to be synthesized to obtain an annotation label. For the specific process, please refer to Figure 3 The content of the examples will not be repeated here.

[0217] After the above steps, the electronic device can obtain the phoneme and label sequence, and input it into the second model or the third model to obtain the target audio.

[0218] At this point, the scheme provided by the embodiment of the present disclosure can obtain the first training data and the second training data, the first training data is used to train the first preset model, and the second training data is used to train the second preset model; then, the first preset model is trained using the first training data to obtain the first model; and the second preset model is trained using the second training data to obtain the front-end model; then, a target model is constructed based on the first model; finally, a speech synthesis model is constructed based on the target model and the front-end model; the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio. In this way, the front-end model in this embodiment can pre-process the text to be recognized and the target model can convert the pre-processed text into the target audio, which can be applied to business scenarios that require speech and improve the user experience.

[0219] Based on the above-mentioned training method of a speech synthesis model, the present disclosure also provides a speech synthesis method, see Fig.16 , the method comprising:

[0220] In step 161, the text to be synthesized is obtained;

[0221] In step 162, the text to be synthesized is input into a preset speech synthesis model for speech synthesis, and a target audio is output;

[0222] The speech synthesis model is constructed based on the target model and the front-end model. The target model is constructed based on a first model obtained by training a first preset model using first training data; the front-end model is a front-end model obtained by training a second preset model using second training data.

[0223] In one embodiment, the first training data includes first original data and its corresponding annotation labels; the first original data includes first original text and speech data generated based on the first original text, and the annotation labels corresponding to the first original data include at least one of the following: a tone label, a rhythm label, a personification label, and a sentiment classification label.

[0224] In one embodiment, the first original text includes a first text, a second text and a third text; the first text is a text in which the number of phonemes exceeds a phoneme number threshold, the second text is a modal particle text, and the third text is a text that represents the requirements of anthropomorphic abnormal pronunciation.

[0225] In one embodiment, the structure of the first preset model includes: a text encoding module, a phoneme filtering module, an addition module, a phoneme classification module, a duration prediction module, an attention mechanism module and an acoustic decoding module; the phoneme filtering module is respectively connected to the text encoding module, the addition module, the phoneme classification module and the duration prediction module; the addition module is respectively connected to the second to last hidden layer in the duration prediction module, the acoustic decoding module and the attention mechanism module;

[0226] The input data of the text encoding module is the first input data; the first input data refers to the data obtained by sorting the preprocessed data and marking the labels in a set order;

[0227] The input data of the attention mechanism module also includes fourth input data; the fourth input data refers to the Mel magnitude spectrum parameter of the preprocessed data;

[0228] The output data of the acoustic decoding module is the predicted acoustic parameter, and the predicted acoustic parameter is used as the first output data;

[0229] The output data of the duration prediction module is the predicted phoneme duration, and the predicted phoneme duration is used as the second output data;

[0230] The output data of the attention mechanism module is a prediction alignment matrix, and the prediction alignment matrix is ​​used as the third output data;

[0231] The output data of the phoneme classification module is phoneme classification data, and the phoneme classification data is used as fourth output data.

[0232] In one embodiment, the second training data includes second original data and corresponding label data; the second original data includes second original text and speech data generated according to the second original text, and the label data corresponding to the second original data includes at least one of the following: a classification label, a type label, a speech speed label, and an abnormal pronunciation label;

[0233] The classification label includes at least one of the following: an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label, a sentence type classification label or an emotion type classification label; the type label includes at least one of the following: a connected reading type label, a repetition type label and a reverse reading type label; the abnormal pronunciation label includes at least one of the following: a dragged sound label, a swallowed sound label and a lazy sound label.

[0234] It should be noted that Fig.16 The training method of the speech synthesis model in the speech synthesis method in the embodiment shown can be referred to Figure 1 The contents of the speech synthesis model in the training method of the speech synthesis model shown will not be repeated here.

[0235] Based on the training method of a speech synthesis model provided in the above embodiment, the present disclosure also provides a training device for a speech synthesis model, see Fig.17 , the device comprises:

[0236] The training data acquisition module 171 is configured to acquire first training data and second training data, wherein the first training data is used to train a first preset model, and the second training data is used to train a second preset model;

[0237] A first model acquisition module 172 is configured to train the first preset model using the first training data to obtain a first model;

[0238] A front-end model acquisition module 173 is configured to train the second preset model using the second training data to obtain a front-end model;

[0239] A target model building module 174 is configured to build a target model based on the first model;

[0240] The speech module construction module 175 is configured to construct a speech synthesis model based on the target model and the front-end model; the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio.

[0241] It should be noted that the device shown in this embodiment matches the contents of the method embodiment, and reference may be made to the contents of the above method embodiment, which will not be repeated here.

[0242] Based on the speech synthesis method provided in the above embodiment, the present disclosure also provides a speech synthesis device, the device comprising:

[0243] A text acquisition module, configured to acquire the text to be synthesized;

[0244] An audio acquisition module is configured to input the text to be synthesized into a preset speech synthesis model for speech synthesis and output a target audio;

[0245] The speech synthesis model is constructed based on a target model and a front-end model. The target model is constructed based on a first model obtained by training a first preset model using first training data; the front-end model is a front-end model obtained by training a second preset model using second training data.

[0246] It should be noted that the device shown in this embodiment matches the content of the speech synthesis method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0247] Fig.18 1 is a block diagram of a mobile terminal according to an exemplary embodiment. For example, the mobile terminal 1800 may be a smart phone, a computer, a digital broadcast terminal, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0248] Reference Fig.18 The mobile terminal 1800 may include one or more of the following components: a processing component 1802 , a memory 1804 , a power component 1806 , a multimedia component 1808 , an audio component 1810 , an input / output (I / O) interface 1812 , a sensor component 1814 , a communication component 1816 , and an image acquisition component 1818 .

[0249] The processing component 1802 generally controls the overall operation of the mobile terminal 1800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1802 may include one or more processors 1820 to execute computer programs. In addition, the processing component 1802 may include one or more modules to facilitate interaction between the processing component 1802 and other components. For example, the processing component 1802 may include a multimedia module to facilitate interaction between the multimedia component 1808 and the processing component 1802.

[0250] The memory 1804 is configured to store various types of data to support the operation of the mobile terminal 1800. Examples of such data include computer programs for any application or method operating on the mobile terminal 1800, contact data, phone book data, messages, pictures, videos, etc. The memory 1804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0251] The power supply component 1806 provides power to various components of the mobile terminal 1800. The power supply component 1806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the mobile terminal 1800. The power supply component 1806 may include a power supply chip, and the controller may communicate with the power supply chip to control the power supply chip to turn on or off a switch device, so that the battery supplies power to the mainboard circuit or not.

[0252] The multimedia component 1808 includes a screen that provides an output interface between the mobile terminal 1800 and the target object. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input information from the target object. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0253] The audio component 1810 is configured to output and / or input audio file information. For example, the audio component 1810 includes a microphone (MIC), and when the mobile terminal 1800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio file information. The received audio file information can be further stored in the memory 1804 or sent via the communication component 1816. In some embodiments, the audio component 1810 also includes a speaker for outputting audio file information.

[0254] The I / O interface 1812 provides an interface between the processing component 1802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, etc.

[0255] The sensor component 1814 includes one or more sensors for providing various aspects of status assessment for the mobile terminal 1800. For example, the sensor component 1814 can detect the open / closed state of the mobile terminal 1800, the relative positioning of components, such as the display screen and keypad of the mobile terminal 1800, and the sensor component 1814 can also detect the position change of the mobile terminal 1800 or a component, the presence or absence of contact between the target object and the mobile terminal 1800, the orientation or acceleration / deceleration of the mobile terminal 1800, and the temperature change of the mobile terminal 1800. In this example, the sensor component 1814 may include a magnetic sensor, a gyroscope, and a magnetic field sensor, wherein the magnetic field sensor includes at least one of the following: a Hall sensor, a thin film magnetoresistive sensor, and a magnetic liquid acceleration sensor.

[0256] The communication component 1816 is configured to facilitate wired or wireless communication between the mobile terminal 1800 and other devices. The mobile terminal 1800 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 1816 receives broadcast information or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0257] In an exemplary embodiment, the mobile terminal 1800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0258] In an exemplary embodiment, there is also provided an electronic device, comprising:

[0259] processor;

[0260] a memory for storing a computer program executable by the processor;

[0261] The processor is configured to execute the computer program in the memory to implement the following Figures 1 to 16 The method of the illustrated embodiment.

[0262] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program, and the executable computer program can be executed by a processor to implement the following. Figures 1 to 16The method of the embodiment shown. The readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0263] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0264] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a speech synthesis model, characterized in that: The method comprises: Acquire first training data and second training data, where the first training data is used to train a first preset model, and the second training data is used to train a second preset model; The first preset model is trained using the first training data to obtain a first model; the structure of the first preset model includes: an acoustic decoding module, a duration prediction module, an attention mechanism module and a phoneme classification module, the output data of the acoustic decoding module is a predicted acoustic parameter, the output data of the duration prediction module is a predicted phoneme duration, the output data of the attention mechanism module is a predicted alignment matrix, and the output data of the phoneme classification module is phoneme classification data; Using the second training data to train the second preset model to obtain a front-end model; Building a target model based on the first model; A speech synthesis model is constructed based on the target model and the front-end model, wherein the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio.

2. The method according to claim 1, characterized in that The first training data includes first original data and its corresponding annotation labels; the first original data includes first original text and speech data generated according to the first original text, and the annotation labels corresponding to the first original data include at least one of the following: tone labels, rhythm labels, personification labels and emotion classification labels.

3. The method according to claim 1 or 2, characterized in that: Obtaining the first training data includes: Acquire a first original text and voice data corresponding to the first original text to obtain first original data; Annotate the first original text according to the voice data corresponding to the first original text to obtain an annotation label corresponding to the first original text; The first original data and its corresponding annotated labels are used as first training data.

4. The method according to claim 3, characterized in that The first original text includes a first text, a second text and a third text; the first text is a text in which the number of phonemes exceeds a phoneme number threshold, the second text is a modal particle text, and the third text is a text that represents the requirements of anthropomorphic abnormal pronunciation.

5. The method according to claim 1, characterized in that Using the first training data to train the first preset model to obtain a first model includes: Preprocessing the first training data to obtain preprocessed data; Based on the annotated labels in the first training data and the preprocessed data, the first input data of the first preset model is obtained; and the second input data, the third input data and the fourth input data of the first preset model are obtained according to the preprocessed data; the first input data refers to the data obtained by sorting the preprocessed data and the annotated labels in a set order; the second input data refers to the data obtained by calculating the Log logarithm of the number of phoneme pronunciation frames in the preprocessed data; the third input data refers to the target alignment matrix converted from the number of phoneme pronunciation frames; the fourth input data refers to the Mel-amplitude spectrum parameters of the preprocessed data; Inputting the first input data into the first preset model to obtain first output data, second output data, third output data and fourth output data output by the first preset model; Obtaining a loss value of a loss function corresponding to the first preset model based on the first input data, the second input data, the third input data, the fourth input data, the first output data, the second output data, the third output data, and the fourth output data; When the loss value exceeds a preset first loss value threshold, the parameters of the first preset model are adjusted according to a preset adjustment method until the loss value is less than the first loss value threshold and the training is stopped to obtain the first model.

6. The method according to claim 5, characterized in that Acquiring first input data of the first preset model based on the annotated labels in the first training data and the preprocessed data includes: The pre-processed data and the marked labels are arranged in a set order to form first input data.

7. The method according to claim 5, characterized in that Acquiring second input data, third input data, and fourth input data of the first preset model according to the preprocessed data, comprising: Calculating the logarithm of the number of phoneme pronunciation frames in the preprocessed data to obtain second input data; Convert the phoneme pronunciation frame number into a target alignment matrix to obtain third input data; Acquire Mel-amplitude spectrum parameters in the preprocessed data to obtain fourth input data.

8. The method according to claim 5, characterized in that Obtaining a loss value of a loss function corresponding to the first preset model based on the first input data, the second input data, the third input data, the fourth input data, the first output data, the second output data, the third output data, and the fourth output data, comprising: Calculate a phoneme classification prediction loss value according to the first input data and the fourth output data, and use the phoneme classification prediction loss value as a first loss value; Calculate a phoneme duration prediction loss value according to the second input data and the second output data, and use the phoneme duration prediction loss value as a second loss value; Calculate an alignment prediction loss value according to the third input data and the third output data, and use the alignment prediction loss value as a third loss value; Calculating an acoustic parameter predicted loss value according to the fourth input data and the first output data, and using the acoustic parameter predicted loss value as a fourth loss value; A loss value of the loss function is calculated based on the first loss value, the second loss value, the third loss value, the fourth loss value, and respective weight values.

9. The method according to claim 8, characterized in that The weight values ​​of the first loss value, the second loss value, the third loss value and the fourth loss value are respectively set based on the following principles: The product of the first loss value and its weight value, the product of the second loss value and its weight value, the product of the third loss value and its weight value, and the product of the fourth loss value and its weight value, the sizes of each product are within the same preset range; The weight values ​​of the fourth loss value, the second loss value, the third loss value and the first loss value decrease in sequence, and a ratio of two adjacent weight values ​​exceeds a set value.

10. The method according to claim 5, characterized in that The structure of the first preset model further includes: a text encoding module, a phoneme filtering module and an addition module; the phoneme filtering module is respectively connected to the text encoding module, the addition module, the phoneme classification module and the duration prediction module; the addition module is respectively connected to the second to last hidden layer in the duration prediction module, the acoustic decoding module and the attention mechanism module; The input data of the text encoding module is the first input data; The input data of the attention mechanism module also includes fourth input data; The acoustic decoding module uses the predicted acoustic parameter as first output data; The duration prediction module uses the predicted phoneme duration as second output data; The attention mechanism module uses the predicted alignment matrix as third output data; The phoneme classification module uses the phoneme classification data as fourth output data.

11. The method according to claim 10, characterized in that The target model includes a second model, and constructing the target model based on the first model includes: Adjust the structure of the first model according to a first preset method to obtain the structure of the second model; Migrating parameter data of the first model to the second model; the second model is used to output the target audio; The first preset method includes: removing the phoneme classification module in the first model; retaining the first output in the first model; and using the output data of the acoustic decoding module in the first model as input data of the attention mechanism module in the first model.

12. The method according to claim 10, characterized in that The target model includes a third model, and constructing the target model based on the first model includes: Adjust the structure of the first model according to a second preset method to obtain the structure of a third model; Migrating the parameter data of the first model to the third model; the third model is used to output the target audio; The second preset method includes: removing the phoneme classification module in the first model; retaining the first output in the first model; replacing the attention mechanism module in the first model with a phoneme expansion module; and using the output data of the acoustic decoding module in the first model as input data of the phoneme expansion module.

13. The method according to claim 1, characterized in that The second training data includes second original data and corresponding label data; the second original data includes second original text and speech data generated according to the second original text, and the label data corresponding to the second original data includes at least one of the following: a classification label, a type label, a speech speed label, and an abnormal pronunciation label; The classification label includes at least one of the following: an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label, a sentence type classification label or an emotion type classification label; the type label includes at least one of the following: a connected reading type label, a repetition type label and a reverse reading type label; the abnormal pronunciation label includes at least one of the following: a dragged sound label, a swallowed sound label and a lazy sound label.

14. The method according to claim 1 or 13, characterized in that: The second training data is obtained, including: Acquire a second original text and voice data corresponding to the second original text to obtain second original data; The second original data is processed to obtain label data corresponding to the second original data; and the second original data and the label data are used as the second training data.

15. The method according to claim 14, characterized in that Processing the second original data to obtain label data corresponding to the second original data includes: The second original text is classified by a preset classification method to obtain a classification label; the classification label includes at least one of the following: an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label, a sentence type classification label or an emotion type classification label; or, The second original text is analyzed by using a context feature analysis method to obtain a type label; the type label includes at least one of the following: a continuous reading type label, a repeated type label, and a reverse reading type label; or, The second original text and its corresponding speech data are aligned by a forced alignment method to obtain the pronunciation duration of each syllable; a speech speed label is generated based on the pronunciation duration of each syllable, and an abnormal pronunciation label is generated based on the pronunciation duration and energy parameter of each syllable; the abnormal pronunciation label includes at least one of the following: a dragging sound label, a swallowing sound label, and a lazy sound label; The classification label, the type label, the speech speed label and the abnormal pronunciation label are used as label data corresponding to the second original data.

16. The method according to claim 1, characterized in that Using the second training data to train the second preset model to obtain a front-end model includes: Inputting the second original text in the second training data into a second preset model, and obtaining predicted label data output by the second preset model; Obtaining a loss value of a loss function corresponding to the second preset model based on the label data corresponding to the second original text and the predicted label data; When the loss value exceeds a preset second loss value threshold, the parameters of the second preset model are adjusted according to a preset adjustment method until the loss value is less than the second loss value threshold and the training is stopped to obtain the front-end model.

17. A speech synthesis method, characterized in that: The method comprises: Get the text to be synthesized; Inputting the text to be synthesized into a preset speech synthesis model for speech synthesis, and outputting target audio; The speech synthesis model is constructed based on a target model and a front-end model, wherein the target model is constructed based on a first model obtained by training a first preset model using first training data; the front-end model is a front-end model obtained by training a second preset model using second training data; the structure of the first preset model includes: an acoustic decoding module, a duration prediction module, an attention mechanism module and a phoneme classification module, the output data of the acoustic decoding module is a predicted acoustic parameter, the output data of the duration prediction module is a predicted phoneme duration, the output data of the attention mechanism module is a predicted alignment matrix, and the output data of the phoneme classification module is phoneme classification data.

18. The method according to claim 17, characterized in that The first training data includes first original data and its corresponding annotation labels; the first original data includes first original text and speech data generated according to the first original text, and the annotation labels corresponding to the first original data include at least one of the following: tone labels, rhythm labels, personification labels and emotion classification labels.

19. The method according to claim 18, characterized in that The first original text includes a first text, a second text and a third text; the first text is a text in which the number of phonemes exceeds a phoneme number threshold, the second text is a modal particle text, and the third text is a text that represents the requirements of anthropomorphic abnormal pronunciation.

20. The method according to claim 17, characterized in that The structure of the first preset model further includes: a text encoding module, a phoneme filtering module and an addition module; the phoneme filtering module is respectively connected to the text encoding module, the addition module, the phoneme classification module and the duration prediction module; the addition module is respectively connected to the second to last hidden layer in the duration prediction module, the acoustic decoding module and the attention mechanism module; The input data of the text encoding module is the first input data; the first input data refers to the data obtained by sorting the preprocessed data and marking the labels in a set order; The input data of the attention mechanism module also includes fourth input data; the fourth input data refers to the Mel magnitude spectrum parameter of the preprocessed data; The acoustic decoding module uses the predicted acoustic parameter as first output data; The duration prediction module uses the predicted phoneme duration as second output data; The attention mechanism module uses the predicted alignment matrix as third output data; The phoneme classification module uses the phoneme classification data as fourth output data.

21. The method according to claim 17, characterized in that The second training data includes second original data and corresponding label data; the second original data includes second original text and speech data generated according to the second original text, and the label data corresponding to the second original data includes at least one of the following: a classification label, a type label, a speech speed label, and an abnormal pronunciation label; The classification label includes at least one of the following: an interjection pronunciation information label, an onomatopoeia / non-onomatopoeia classification label, a sentence type classification label or an emotion type classification label; the type label includes at least one of the following: a connected reading type label, a repetition type label and a reverse reading type label; the abnormal pronunciation label includes at least one of the following: a dragged sound label, a swallowed sound label and a lazy sound label.

22. A training device for a speech synthesis model, characterized in that: The device comprises: A training data acquisition module is configured to acquire first training data and second training data, wherein the first training data is used to train a first preset model, and the second training data is used to train a second preset model; A first model acquisition module is configured to train the first preset model using the first training data to obtain a first model; the structure of the first preset model includes: an acoustic decoding module, a duration prediction module, an attention mechanism module and a phoneme classification module, the output data of the acoustic decoding module is a predicted acoustic parameter, the output data of the duration prediction module is a predicted phoneme duration, the output data of the attention mechanism module is a predicted alignment matrix, and the output data of the phoneme classification module is phoneme classification data; A front-end model acquisition module, configured to train the second preset model using the second training data to obtain a front-end model; A target model building module, configured to build a target model based on the first model; The speech module construction module is configured to construct a speech synthesis model based on the target model and the front-end model; the speech synthesis model is used to perform speech synthesis on the text to be synthesized to obtain the target audio.

23. A speech synthesis device, characterized in that: The device comprises: A text acquisition module, configured to acquire the text to be synthesized; An audio acquisition module is configured to input the text to be synthesized into a preset speech synthesis model for speech synthesis and output a target audio; The speech synthesis model is constructed based on a target model and a front-end model, wherein the target model is constructed based on a first model obtained by training a first preset model using first training data; the front-end model is a front-end model obtained by training a second preset model using second training data; the structure of the first preset model includes: an acoustic decoding module, a duration prediction module, an attention mechanism module and a phoneme classification module, the output data of the acoustic decoding module is a predicted acoustic parameter, the output data of the duration prediction module is a predicted phoneme duration, the output data of the attention mechanism module is a predicted alignment matrix, and the output data of the phoneme classification module is phoneme classification data.

24. An electronic device, characterized in that: include: processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 21.

25. A computer-readable storage medium, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 21 can be implemented.

Images