A speech synthesis method and device with natural pause processing, electronic equipment and computer readable medium

Abstract

The application discloses a speech synthesis method with natural pause processing, comprising the following steps: extracting punctuation mark, part-of-speech mark, word mark and phoneme sequence from original text, respectively encoding and fusing into input information; the punctuation mark is formed by marking different labels with different marks and points in the original text; extracting first hidden layer characteristic variable according to the input information; performing phoneme duration prediction, energy prediction and pitch prediction according to the first hidden layer characteristic variable, and fusing to obtain second hidden layer characteristic variable; decoding to obtain mel spectrum variable according to the second hidden layer characteristic variable; synthesizing speech corresponding to the original text according to the mel spectrum variable. The application also discloses a speech synthesis device with natural pause processing, electronic equipment and computer readable medium. The application can solve the problems of unclear pause position, monotonous rhythm and harshness in the prior art.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech synthesis, and particularly relates to a speech synthesis method and device with natural pause processing, an electronic device and a computer readable medium. BACKGROUND

[0002] With the development of deep learning, speech synthesis (TTS) systems based on neural networks have made breakthrough progress and applications, and are widely used in virtual assistants, intelligent reading, intelligent navigation and other fields. As an official language of China, Mandarin has important research and application value. However, due to the complexity and diversity of Chinese sentence structure, the speech synthesis of Mandarin is still not ideal in naturalness and expressiveness, especially the pause processing in the speech is one of the keys to affect the naturalness of speech and semantic understanding.

[0003] The pause of a sentence can be divided into two ways. One is the pause between sentences. In modern Chinese, different punctuation marks are mainly used to divide sentences, and different punctuation marks represent different pause times. For example, the pause times represented by the colon, comma and period are successively longer, and the length of the pause interval between short sentences can reflect the rhythm of the sentence, which has an important influence on the naturalness of speech. The other is the pause between phrases in a sentence. Since there is no explicit symbol marking between phrases, some natural language processing related techniques are needed to mark the features so that the TTS model can learn the pause position and duration.

[0004] In the prior art, for the punctuation marks between sentences or the punctuation marks within a sentence, the existing speech synthesis model usually uses a global same single label to mark the punctuation marks, and then trains the data input TTS model to realize the pause of the synthesized speech at the punctuation marks. However, this cannot reflect the different pause times of different punctuation marks in natural language, and the rhythm of the synthesized Mandarin speech is monotonous and harsh. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application provides a speech synthesis method, device, electronic device and computer readable medium with natural pause processing, which can solve the problems of non-obvious pause position, monotonous rhythm and harshness of synthesized speech in the prior art.

[0006] In a first aspect, a speech synthesis method with natural pause processing comprises the following steps:

[0007] Extracting punctuation mark labels, part-of-speech labels, word labels and phoneme sequences from the original text, respectively encoding and fusing them into input information; the punctuation mark labels are formed by marking different marks and dots in the original text as different labels, respectively;

[0008] extracting a first hidden layer feature variable according to the input information;

[0009] performing phoneme duration prediction, energy prediction and pitch prediction according to the first hidden layer feature variable, and fusing to obtain a second hidden layer feature variable;

[0010] decoding according to the second hidden layer feature variable to obtain a mel spectrum variable;

[0011] synthesizing speech corresponding to the original text according to the mel spectrum variable.

[0012] As a preferred solution, the punctuation mark label, the part-of-speech label, the word label and the phoneme sequence are extracted from the original text, respectively encoded and fused into input information, including the following steps:

[0013] extracting punctuation mark labels from the original text and converting them into embedding vectors;

[0014] extracting part-of-speech labels from the original text and converting them into embedding vectors;

[0015] extracting word labels from the original text and converting them into embedding vectors;

[0016] convert Chinese characters and numbers in the original text into phonemes and encode them into text sequences;

[0017] fuse the embedding vectors corresponding to the punctuation mark labels, the part-of-speech labels and the word labels with the text sequences to obtain the input information.

[0018] As a preferred solution, the word label is formed by splitting each word in the original text into phonemes and marking them in order.

[0019] As a preferred solution, the first hidden layer feature variable is extracted according to the input information, including the following steps:

[0020] encode the context information of the input information according to the multi-head attention mechanism, and perform residual connection and regularization processing;

[0021] extract and fuse hidden layer features of different lengths of phrases through multiple convolution kernels of different sizes, and perform residual connection and regularization processing to obtain the first hidden layer feature variable.

[0022] As a preferred solution, the second hidden layer feature variable is obtained by performing phoneme duration prediction, energy prediction and pitch prediction according to the first hidden layer feature variable, including the following steps:

[0023] phoneme duration prediction is performed according to the first hidden layer feature variable to obtain duration information of each phoneme and residual fusion is performed by a length regulator;

[0024] energy prediction is performed to obtain volume information of each phoneme and residual fusion is performed;

[0025] pitch prediction is performed to obtain pitch information of each phoneme and residual fusion is performed.

[0026] As a preferred solution, after the first hidden layer feature variable is extracted according to the input information, the following steps are further included:

[0027] phrase duration prediction is performed according to the first hidden layer feature variable to obtain duration information of each phrase;

[0028] loss is calculated according to the sum of phoneme durations in a phrase predicted by the phoneme duration prediction module and the phrase duration predicted by the phrase duration prediction module.

[0029] As a preferred solution, the decoding of the second hidden layer feature variable to obtain a mel-frequency spectrum variable includes the following steps:

[0030] The second hidden layer feature variable is encoded according to a multi-head attention mechanism, residual connection and regularization processing are performed;

[0031] Different lengths of acoustic features are extracted and fused into hidden layer features by multiple convolution kernels of different sizes, and residual connection and regularization processing are performed to obtain corresponding mel-frequency spectrum variables.

[0032] In a second aspect, a speech synthesis device with natural pause processing includes:

[0033] A front-end processing module is configured to extract punctuation marks, part-of-speech marks, word marks and phoneme sequences from original text, and to encode and fuse the marks into input information;

[0034] An encoder module is configured to extract a first hidden layer feature variable according to the input information;

[0035] A variable prediction module is configured to perform phoneme duration prediction, energy prediction and pitch prediction according to the first hidden layer feature variable, and to fuse to obtain a second hidden layer feature variable;

[0036] A decoder module is configured to decode the second hidden layer feature variable to obtain a mel-frequency spectrum variable;

[0037] A vocoder module is configured to synthesize speech corresponding to the original text according to the mel-frequency spectrum variable.

[0038] In a third aspect, an electronic device includes:

[0039] One or more processors;

[0040] A storage device on which one or more programs are stored;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the speech synthesis methods with natural pause processing as described in the first aspect.

[0042] Fourthly, a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the speech synthesis methods with natural pause processing as described in the first aspect.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. By marking different punctuation marks and symbols as different labels, and combining implicit part-of-speech tags and character tags, hidden features are extracted after fusion, and then phoneme duration, energy and pitch information are fused to establish a Mandarin TTS system with natural pause processing. This embodiment can effectively solve the problems of unclear pause positions and single global pause duration in Mandarin synthesized speech.

[0045] 2. By employing a multi-head attention mechanism combined with multi-scale convolution during the encoding and decoding processes, local features of text can be utilized more effectively to extract features from characters to words, phrases, sentences, and paragraphs, thereby improving the natural pauses between phrases.

[0046] 3. By combining phoneme duration prediction with energy prediction and pitch prediction in variable prediction, the speech synthesis effect can be further improved.

[0047] 4. By incorporating phrase duration prediction into variable prediction to establish an error correction mechanism with phoneme duration prediction, the accuracy of the model's phrase duration prediction can be effectively improved, thereby enhancing the overall robustness of the model's prediction. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a speech synthesis method with natural pause processing according to the present invention.

[0050] Figure 2 Flowchart of step S1 of the speech synthesis method with natural pause processing of the present application;

[0051] Figure 3 Flowchart of step S2 of the speech synthesis method with natural pause processing of the present application;

[0052] Figure 4 Method diagram of multi-scale convolution in step S2 of the speech synthesis method with natural pause processing of the present application;

[0053] Figure 5 Flowchart of step S3 of the speech synthesis method with natural pause processing of the present application;

[0054] Figure 6 Flowchart of error correction for phrase duration prediction in the speech synthesis method with natural pause processing of the present application;

[0055] Figure 7 Flowchart of step S4 of the speech synthesis method with natural pause processing of the present application;

[0056] Figure 8 Audio effect diagram generated by the prior art in Example 1;

[0057] Figure 9 Audio effect diagram generated by the present application in Example 1;

[0058] Figure 10 Audio effect diagram generated by the prior art in Example 2;

[0059] Figure 11 Audio effect diagram generated by the present application in Example 2;

[0060] Figure 12 Structure diagram of the speech synthesis device with natural pause processing of the present application;

[0061] Figure 13 Structure diagram of the front-end processing module of the speech synthesis device with natural pause processing of the present application;

[0062] Figure 14 Structure diagram of the encoder module of the speech synthesis device with natural pause processing of the present application;

[0063] Figure 15 Structure diagram of the variable prediction module of the speech synthesis device with natural pause processing of the present application;

[0064] Figure 16A schematic diagram of an end-to-end acoustic model structure of a speech synthesis device with natural pause processing according to the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0067] In a first aspect, a speech synthesis method with natural pause processing comprises the following steps as shown in the figure: Figure 1

[0068] S1: front-end processing: extracting punctuation mark labels, part-of-speech labels, word labels and phoneme sequences from the original text, and fusing them into input information after encoding; the punctuation mark labels are formed by marking different labels with different marks and points in the original text;

[0069] S2: encoding: extracting first hidden layer feature variables according to the input information;

[0070] S3: variable prediction: phoneme duration prediction, energy prediction and pitch prediction are performed according to the first hidden layer feature variables, and second hidden layer feature variables are fused;

[0071] S4: decoding: obtaining mel spectrum variables by decoding according to the second hidden layer feature variables;

[0072] S5: synthesis: synthesizing speech corresponding to the original text according to the mel spectrum variables.

[0073] In this embodiment, an end-to-end acoustic model is used, and its main structure is derived from the FastSpeech2 structure. It is a non-autoregressive speech synthesis network with the advantages of parallel operation and processing long-distance dependencies, mainly including an encoder, a variable predictor and a decoder corresponding to steps S2-S4. The synthesis in step S5 uses a vocoder, which is prior art and will not be described here.

[0074] ​The punctuation mark label can be marked by a designed <“punctuation mark”:“label”> dictionary for punctuation marks in a text sentence. In a Chinese sentence, there are 16 commonly used punctuation marks, which can be divided into two types of point marks and label marks. There are 7 point marks, including comma, colon, semicolon, colon, period, question mark and exclamation mark, which represent different types of pauses. There are 9 label marks, including quotation marks, parentheses, dashes, ellipses, emphasis marks, connecting marks, spacing marks, book title marks and proper name marks. The inventors consider that in the current Mandarin speech synthesis system, the punctuation marks between sentences or within sentences are marked by a single label globally, and then trained with the data input TTS model to realize the pause of the synthesized speech at the punctuation mark. The inventors find that this scheme has limitations: since the punctuation marks are marked by a single label globally, the influence of different punctuation marks on the pause time is not considered, which leads to the fact that the pause time between all sentences predicted by the TTS model is basically unchanged, and the characteristics of different punctuation marks with different pause times cannot be reflected, making the synthesized Mandarin speech rhythm monotonous and harsh. Considering that although the label mainly indicates the nature and function of the sentence, people will also have a certain degree of pause when reading Chinese sentences when encountering label marks. Moreover, the existing speech synthesis model does not mark the label marks to realize the pause. Therefore, in the embodiment, the speech pauses caused by the above-mentioned Chinese punctuation marks are marked as different labels according to the different types of punctuation marks. The specific punctuation mark dictionary can be shown in Table 1 (emphasis marks are generally marked below the characters, which are not considered here).

[0075] Table 1 Punctuation mark dictionary

[0076]

[0077] The part-of-speech label has been widely used in the field of natural language processing. In the front-end processing in the application, an algorithm can be used to perform word segmentation on the text and then label the part-of-speech.

[0078] The word label can be performed on each word group after word segmentation of the text. Generally, a label set {S, B, M, E} can be defined, S represents a single word as a word, B represents the first word of a word, M represents the middle word of a word, and E represents the last word of a word. For example, “full of hope in 2022” can be marked as: [BE BE S BMMME].

[0079] The punctuation mark label is an explicit label feature, and the part-of-speech label and the word label are implicit label features.

[0080] The embodiment is based on the characteristics of Chinese language, and by designing explicit and implicit label features and combining feature fusion and extraction mechanism, a Mandarin TTS system with natural pause processing is established, which can effectively solve the problems of unclear pause position and single global pause duration in Mandarin synthesized speech.

[0081] A preferred embodiment, the punctuation mark, part-of-speech tag, word tag and phoneme sequence are extracted from the original text, and are fused into input information after encoding, as shown in Figure 2 The method comprises the following steps:

[0082] S101: extracting punctuation mark from the original text and converting it into embedded vector;

[0083] S102: extracting part-of-speech tag from the original text and converting it into embedded vector;

[0084] S103: extracting word tag from the original text and converting it into embedded vector;

[0085] S104: converting Chinese characters and numbers in the original text into phonemes and encoding them into text sequence;

[0086] S105: fusing the embedded vectors corresponding to the punctuation mark, part-of-speech tag and word tag with the text sequence to obtain the input information.

[0087] In the embodiment, as shown in the figure, the original text information is input into different modules to extract punctuation mark, part-of-speech tag, word tag and phoneme sequence respectively. As shown in Table 1 above, if only the above-mentioned number coding is used, not only it is difficult to perform efficient matrix calculation, but also the single label cannot represent the relationship between each punctuation mark, so it is necessary to perform symbol embedding operation on the punctuation mark to convert it into embedded vector. Here, symbol embedding operation is also needed for part-of-speech tag and word tag to convert them into embedded vector.

[0088] As a further optimized embodiment, the word tag extracted from the original text can be phoneme splitting and sequentially labeled for each word in the original text.

[0089] Considering that the input of the TTS model is a phoneme sequence, the phoneme sequence length is greater than the word token sequence, and if only the word token sequence is used as a feature, it also needs to be expanded according to the phoneme length. The word token can adopt fine-grained word token, that is, each word is marked according to the phoneme arrangement order. The phonemes in the above example "full of hope in 2022" are "ch ong1 man3 x i1 w uang4 d e5 er4 l ing2 er4 er4 n ian2", and the fine-grained word token can be expressed as [B1B2E1E2 B1B2E1E2 S1S2 B1M1M2M3M4E1E2]. The fine-grained word token length is consistent with the phoneme length, and contains boundary and order information.

[0090] A preferred embodiment, the first hidden layer feature variable is extracted according to the input information, such as Figure 3 As shown, comprising the following steps:

[0091] S201: Multi-head attention mechanism: according to the multi-head attention mechanism, the context information of the input information is encoded, and residual connection and regularization processing are performed;

[0092] S202: Multi-scale convolution: different length phrase features are extracted and fused into hidden layer features by multiple different size convolution kernels, and residual connection and regularization processing are performed to obtain the first hidden layer feature variable.

[0093] In this embodiment, the multi-head attention mechanism belongs to the prior art, which will not be described here. After the multi-head attention mechanism, we propose a multi-scale filter structure, which includes multiple one-dimensional convolution channels of different sizes. Different size convolution kernels can cover different ranges of features. According to the characteristics of Chinese sentences, Chinese phrases generally contain 1-4 words, and the words before and after a word are necessarily related to the word itself, so local feature extraction needs to be fully considered. As shown in Figure 4 Taking the case that the filter includes three one-dimensional convolution channels as an example, the filter sizes adopted are n1, n2 and n3 (n1, n2 and n3 are different size convolution kernels), which can realize the extraction of phrase features of different lengths, and finally fuse the three features. As shown in Figure 4 When multiple convolution layers are stacked, the expression (m+2p-h) / s+1 is used, where m is the input sequence length, p is the padding size, h is the filter size, and s is the step length. By adjusting the values of each parameter, the input encoding length D1 and the output encoding length dimension can be kept unchanged.

[0094] After processing by the multi-head attention mechanism or multi-scale convolution, the problem of gradient disappearance in the training process can be alleviated by establishing the residual connection, and the problem of overfitting can be alleviated by the regularization processing.

[0095] By adopting the multi-head attention mechanism combined with multi-scale convolution in the encoding and decoding process, the local features of the text can be more effectively utilized, the feature extraction from the word to the phrase, sentence and paragraph level in the text can be realized, and the natural pause effect between phrases can be effectively improved.

[0096] A preferred embodiment, the phoneme duration prediction, energy prediction and pitch prediction according to the first hidden layer feature variable, fusion to obtain the second hidden layer feature variable, such as Figure 5 As shown, comprising the following steps:

[0097] S301: According to the first hidden layer feature variable, the phoneme duration prediction is carried out, the length information of each phoneme is obtained, and the residual fusion is carried out through the length regulator;

[0098] S302: Energy prediction is carried out, the volume information of each phoneme is obtained, and residual fusion is carried out;

[0099] S303: Perform pitch prediction, obtain the pitch information of each phoneme and perform residual fusion.

[0100] In this embodiment, the pitch, energy and duration of the phoneme are predicted as input hidden layer features, and then the three variables are fused with the hidden layer features as new hidden layer features, i.e. the second hidden layer feature variable. The phoneme duration prediction can be trained as a phoneme duration predictor according to the real phoneme duration as a label to predict the duration of each phoneme.

[0101] The residual fusion refers to the fusion of the hidden layer features before and after the prediction through the residual connection. In particular, the phoneme duration prediction realizes the alignment of phoneme duration and mel spectrum through the length regulator.

[0102] Since speech synthesis is a one-to-many mapping relationship (i.e. multiple spectra correspond to the same phoneme), phoneme duration prediction is needed to align the phoneme and mel spectrum. The linguistic features obtained from the encoder can obtain the duration of each phoneme after phoneme duration prediction. As shown in Figure 16 The linguistic features can be expanded according to the phoneme duration through the length regulator (LR, Length Regular), i.e. the length regulator can be used to expand the phoneme sequence according to the duration of each phoneme to match the length of the mel spectrum, so as to realize the alignment of the linguistic features and the mel spectrum.

[0103] A preferred embodiment, after the first hidden layer feature variable is extracted according to the input information, as shown inFigure 6 as shown, comprising the following steps:

[0104] S311: Perform phrase duration prediction according to the first hidden layer feature variable, and obtain duration information of each phrase;

[0105] S312: Calculate loss according to the sum of phoneme durations in the phrase predicted by the phoneme duration prediction module and the phrase duration predicted by the phrase duration prediction module.

[0106] In this embodiment, the phrase duration prediction can be trained as a label according to the actual pronunciation duration of the phrase to predict the duration of each phrase.

[0107] In this embodiment, mean square error (MSE) can be used as a loss function for training the end-to-end acoustic model. The phrase duration prediction and the phoneme duration prediction together establish an error correction mechanism for the phrase duration. In the training process, the phrase duration prediction takes the actual duration of the phrase as the target value, and the predicted phrase duration is subtracted from the phoneme duration predicted by the phoneme prediction (MSE loss). This can effectively improve the accuracy of the model in predicting the duration of the phrase and improve the robustness of the model in predicting the overall.

[0108] In a preferred embodiment, the decoding according to the second hidden layer feature variable obtains a mel-spectrum variable, such as Figure 7 as shown, comprising the following steps:

[0109] S401: Multi-head attention mechanism: encode the second hidden layer feature variable according to the multi-head attention mechanism, and perform residual connection and regularization processing;

[0110] S402: Multi-scale convolution: extract and fuse acoustic features of different lengths into hidden layer features through multiple convolution kernels of different sizes, and perform residual connection and regularization processing to obtain corresponding mel-spectrum variables.

[0111] In this embodiment, the decoder structure is completely consistent with the encoder structure, except that the parameters of the two are independent, and the multi-scale convolution filter size is controlled by the LR, i.e., the filter value is set according to the length of the phoneme expansion.

[0112] In order to better illustrate the technical effects of the present application, the prior art and the present application will be compared, and specific examples are listed as follows:

[0113] Embodiment 1:

[0114] South China University of Technology is located in Guangzhou, is a direct subordinate of the Ministry of Education, the national key university, campus is divided into five mountains campus, university town campus and Guangzhou International Campus, is the first "national civilized campus" to obtain units.

[0115] The audio effect generated in the prior art is as shown in Figure 8 The audio effect generated in the present application is as shown in Figure 9 .

[0116] From the comparison of the above two figures, it can be found that the duration of the pause at the punctuation mark changes obviously, and the pause is more reasonable.

[0117] Example 2:

[0118] The original text used is: Hello everyone, I am a graduate student from the School of Computer Science and Engineering of South China University of Technology.

[0119] The audio effect generated in the prior art is as shown in Figure 10 The audio effect generated in the present application is as shown in Figure 11 .

[0120] From the comparison of the above two figures, it can be found that the duration of the pause at the punctuation mark changes obviously, and the pause is more reasonable.

[0121] In a second aspect, a speech synthesis device with natural pause processing is shown in Figure 12 , comprising:

[0122] A front-end processing module is used to extract punctuation mark, part-of-speech tag, word tag and phoneme sequence from the original text, and fuse them into input information after encoding; the punctuation mark is marked by different marks and dots in the original text to form different labels;

[0123] An encoder module is used to extract first hidden layer feature variables according to the input information;

[0124] A variable prediction module is used to perform phoneme duration prediction, energy prediction and pitch prediction according to the first hidden layer feature variables, and fuse to obtain second hidden layer feature variables;

[0125] A decoder module is used to decode to obtain mel spectrum variables according to the second hidden layer feature variables;

[0126] A vocoder module is used to synthesize speech corresponding to the original text according to the mel spectrum variables.

[0127] The embodiment is based on the characteristics of Chinese language, and by designing explicit and implicit label features and combining feature fusion and extraction mechanism, a Mandarin TTS system with natural pause processing is established, which can effectively solve the problems of unclear pause position and single global pause duration in Mandarin synthesized speech.

[0128] A preferred embodiment, the front-end processing module, as shown in Figure 13 , comprises:

[0129] The punctuation mark embedding module is used for punctuation mark labeling and converting the original text into an embedding vector.

[0130] The part-of-speech tagging embedding module is used for part-of-speech tagging and converting the original text into an embedding vector.

[0131] The character tagging embedding module is used for character tagging and converting the original text into an embedding vector.

[0132] The phoneme encoding module is used for converting Chinese characters and numbers in the original text into phonemes and encoding them into a text sequence.

[0133] The fusion module is used for fusing the embedding vectors corresponding to the punctuation mark, part-of-speech tag and character tag with the text sequence to obtain input information for the encoder module.

[0134] In particular, the character tagging of the original text can be phoneme splitting and sequential tagging of each character in the original text.

[0135] A preferred embodiment, the encoder module, as shown in Figure 14 , comprises:

[0136] The multi-head attention mechanism module is used for encoding the context information of the input information according to the multi-head attention mechanism, and performing residual connection and regularization processing.

[0137] The multi-scale convolution module is used for extracting and fusing phrase features of different lengths into hidden layer features through multiple convolution kernels of different sizes, and performing residual connection and regularization processing to obtain the first hidden layer feature variable.

[0138] In this embodiment, the multi-head attention mechanism belongs to the prior art and will not be described here. For the encoder module, after the multi-head attention mechanism, we propose a multi-scale filter structure, which includes multiple one-dimensional convolution channels of different sizes. Different size convolution kernels can cover different ranges of features. According to the characteristics of Chinese sentences, a Chinese phrase generally contains 1-4 characters, and the characters before and after a character are necessarily related to the character itself, so local feature extraction needs to be fully considered. For example, as shown in Figure 4 Taking the case where the filter includes three one-dimensional convolution channels, the filter sizes used are n1, n2, and n3 (n1, n2, and n3 are different size convolution kernels), which can realize the extraction of phrase features of different lengths, and finally the three features are fused. As shown in Figure 4 When multiple convolution layers are stacked, the expression (m+2p-h) / s+1 is used, where m is the input sequence length, p is the padding size, h is the filter size, and s is the step size. By adjusting the values of each parameter, the input encoding length D1 and the output encoding length dimension can be kept unchanged.

[0139] As shown in Figure 16 After processing by the multi-head attention mechanism or the multi-scale convolution, the problem of gradient disappearance in the training process can be alleviated by establishing the residual connection, and the problem of overfitting can be alleviated by the regularization processing.

[0140] By using the multi-head attention mechanism combined with multi-scale convolution in the encoding and decoding process, the local features of the characters can be more effectively utilized, the character-to-word, phrase, sentence, and paragraph level feature extraction in the text can be realized, and the natural pause effect between phrases can be effectively improved.

[0141] A preferred embodiment, the variable prediction module, as shown in Figure 15 includes:

[0142] A phoneme duration prediction module is configured to perform phoneme duration prediction based on the first hidden layer feature variable, obtain duration information of each phoneme, and perform residual fusion through a length adjuster.

[0143] An energy prediction module is configured to perform energy prediction, obtain volume information of each phoneme, and perform residual fusion.

[0144] A pitch prediction module is configured to perform pitch prediction, obtain pitch information of each phoneme, and perform residual fusion.

[0145] In this embodiment, the variable predictor module mainly takes the hidden layer feature as input to predict the pitch, energy, and duration of the phoneme, and then fuses the three variables with the hidden layer feature as a new hidden layer feature, i.e., the second hidden layer feature variable.

[0146] The residual fusion refers to fusing the hidden layer features before and after the prediction through a residual connection. In particular, the phoneme duration prediction module realizes the alignment of the phoneme duration and the mel spectrum through a length regulator.

[0147] In the variable predictor, since the speech synthesis is a one-to-many mapping relationship (i.e., multiple spectrums correspond to the same phoneme), the phoneme duration prediction is needed to align the phoneme and the mel spectrum. The linguistic features obtained from the encoder are extended according to the phoneme duration through a phoneme duration prediction, so as to obtain the duration of each phoneme. As shown in Figure 16 , the linguistic features can be aligned with the mel spectrum through a length regulator (LR) which stretches the linguistic features according to the phoneme duration (i.e., the length regulator stretches the phoneme sequence according to the duration of each phoneme to match the length of the mel spectrum), so as to realize the alignment of the linguistic features and the mel spectrum.

[0148] As a further preferred, the variable prediction module, as shown in Figure 15 , further comprises:

[0149] a phrase duration prediction module for performing phrase duration prediction to obtain the duration information of each phrase;

[0150] a loss module for calculating a loss according to the sum of the phoneme durations within the phrase predicted by the phoneme duration prediction module and the phrase duration predicted by the phrase duration prediction module.

[0151] In this embodiment, the loss module can use mean square error (MSE) as a loss function for training the end-to-end acoustic model in this embodiment. The phrase duration prediction module and the phoneme duration prediction module jointly establish an error correction mechanism for the phrase duration. In the training process, the phrase duration prediction takes the real duration of the phrase as a target value, and the predicted phrase duration and the phoneme duration predicted by the phoneme duration prediction (i.e., the sum of the durations of the phonemes within the phrase) are calculated by difference (MSE Loss). This can effectively improve the accuracy of the model in predicting the duration of the phrase and improve the robustness of the model in overall prediction.

[0152] A preferred embodiment, the decoder module, as shown in Figure 16 , has the same structure as the encoder module, comprising a multi-head attention mechanism module and a multi-scale convolution module, wherein the multi-head attention mechanism module performs residual connection and regularization processing after the multi-head attention mechanism, and the multi-scale convolution module performs residual connection and regularization processing after extracting the features. The difference between the decoder module and the encoder module mainly lies in that the input and output of the two are different, and the parameters of the two are independent (i.e., the parameters are different).

[0153] In a third aspect, an electronic device comprises:

[0154] one or more processors;

[0155] a storage device having stored thereon one or more programs;

[0156] when the one or more programs are executed by the one or more processors, cause the one or more processors to carry out the speech synthesis method with natural pause processing of any one of the first aspect.

[0157] In a fourth aspect, a computer readable medium having stored thereon a computer program, wherein the program, when executed by a processor, carries out the speech synthesis method with natural pause processing of any one of the first aspect.

[0158] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as limiting the scope of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application.

Claims

1. A speech synthesis method with natural pause processing, characterized in that, Includes the following steps: Punctuation marks, part-of-speech tags, character tags, and phoneme sequences are extracted from the original text, encoded separately, and then fused into input information; the punctuation marks are formed by marking different numbers and points in the original text as different labels; Extract the first hidden layer feature variables based on the input information; Based on the first hidden layer feature variables, phoneme duration prediction, energy prediction and pitch prediction are performed, and then fused to obtain the second hidden layer feature variables. Mel spectrum variables are obtained by decoding the second hidden layer feature variables; Speech corresponding to the original text is synthesized based on the Mel spectrum variables; The process of extracting punctuation marks, part-of-speech tags, character tags, and phoneme sequences from the original text, encoding them separately, and then fusing them into input information includes the following steps: Extract punctuation marks from the original text and convert them into embedding vectors; Extract part-of-speech tags from the original text and convert them into embedding vectors; Extract word tags from the original text and convert them into embedding vectors; The Chinese characters and numbers in the original text are converted into phonemes and encoded into a text sequence; The embedding vectors corresponding to the punctuation marks, part-of-speech tags, and character tags are fused with the text sequence to obtain the input information; The character markers are formed by splitting each character in the original text according to its phonemes and marking them in sequence.

2. The speech synthesis method with natural pause processing according to claim 1, characterized in that, The step of extracting the first hidden layer feature variables based on the input information includes the following steps: The context information of the input information is encoded according to the multi-head attention mechanism, and residual connections and regularization are performed. The first hidden layer feature variable is obtained by extracting and fusing phrase features of different lengths using multiple convolutional kernels of different sizes, followed by residual connection and regularization.

3. The speech synthesis method with natural pause processing according to claim 1, characterized in that, The step of predicting phoneme duration, energy, and pitch based on the first hidden layer feature variables and fusing them to obtain the second hidden layer feature variables includes the following steps: Phoneme duration prediction is performed based on the first hidden layer feature variables to obtain the duration information of each phoneme, and residual fusion is performed through a length adjuster. Perform energy prediction, obtain volume information for each phoneme, and perform residual fusion; Pitch prediction is performed to obtain pitch information for each phoneme and residual fusion is then performed.

4. The speech synthesis method with natural pause processing according to claim 3, characterized in that, After extracting the first hidden layer feature variables based on the input information, the following steps are also included: Predict phrase duration based on the first hidden layer feature variables to obtain duration information for each phrase; The loss is calculated by combining the sum of the phoneme durations predicted by the phoneme duration prediction module with the phrase duration predicted by the phrase duration prediction module.

5. The speech synthesis method with natural pause processing according to claim 1, characterized in that, The step of decoding the second hidden layer feature variables to obtain the Mel spectrum variables includes the following steps: The second hidden layer feature variables are encoded according to the multi-head attention mechanism, and residual connections and regularization are performed. Acoustic features of different lengths are extracted and fused into hidden layer features by using multiple convolutional kernels of different sizes. Residual connections and regularization are then performed to obtain the corresponding Mel spectrum variables.

6. A speech synthesis device with natural pause processing, characterized in that, include: The front-end processing module is used to extract punctuation marks, part-of-speech tags, character tags, and phoneme sequences from the original text, encode them separately, and then fuse them into input information. The punctuation marks are formed by marking different numbers and points in the original text with different labels; The encoder module is used to extract the first hidden layer feature variables based on the input information; The variable prediction module is used to predict phoneme duration, energy and pitch based on the first hidden layer feature variables, and then fuse them to obtain the second hidden layer feature variables. The decoder module is used to decode the second hidden layer feature variables to obtain the Mel spectrum variables; A vocoder module is used to synthesize speech corresponding to the original text based on the Mel spectrum variables; The front-end processing module includes: The punctuation mark embedding module is used to mark the original text with punctuation marks and convert it into an embedding vector; The part-of-speech tagging embedding module is used to tag the original text with parts of speech and convert it into an embedding vector; The character tag embedding module is used to tag the original text with characters and convert it into an embedding vector; A phoneme encoding module is used to convert Chinese characters and numbers in the original text into phonemes and encode them into a text sequence. The fusion module is used to fuse the embedding vectors corresponding to the punctuation mark tags, part-of-speech tags, and character tags with the text sequence to obtain input information for the encoder module; The word tagging of the original text involves splitting each word in the original text according to its phonemes and tagging them in sequence.

7. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement a speech synthesis method with natural pause processing as described in any one of claims 1-5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements a speech synthesis method with natural pause processing as described in any one of claims 1-5.

Images