A parallel speech synthesis method based on prosodic features, device, medium and equipment

Abstract

The present invention proposes a parallel speech synthesis method and device, equipment, and medium based on prosodic features, which belongs to the field of speech synthesis technology. The method comprises: first regularizing the speech text to be synthesized; converting the regularized speech text into a phoneme sequence and a phoneme-level prosodic sequence in parallel; predicting the mel-spectrogram of the phoneme sequence and the phoneme-level prosodic sequence using an acoustic model; and converting the mel-spectrogram into a speech signal of the speech text to be synthesized using a vocoder. This patent combines the advantages of autoregressive synthesis and non-autoregressive synthesis, which can not only effectively improve the sound quality, fluency, speed, and robustness of speech synthesis, but also realize real-time and efficient speech synthesis. The system developed based on this patent can be widely used in intelligent speech synthesis in the field of artificial intelligence, such as intelligent customer service, intelligent audio, voice broadcasting, map navigation, audiobooks and other human-computer interaction scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular to a parallel speech synthesis method based on prosodic features, a device, a medium and a device. Background Art

[0002] Speech synthesis refers to synthesizing understandable and natural speech from text, also known as text to speech (TTS). It has a wide range of applications in human communication and has always been a hot research topic in the fields of artificial intelligence, natural language and speech processing. Speech synthesis has gone through three stages of development, namely splicing-based speech synthesis, statistical parameter-based speech synthesis and neural network-based speech synthesis. With the vigorous development of deep learning, speech synthesis based on neural networks has developed rapidly, and the sound quality and natural fluency of speech synthesis have been significantly improved. At present, the mainstream speech synthesis technology based on neural networks is divided into two categories: autoregressive speech synthesis and non-autoregressive speech synthesis. Autoregressive speech synthesis has high speech quality and fluency, but due to its autoregressive structure, the synthesis speed is slow, the robustness is low, and there will be problems of repeated words and swallowing words. At present, the method to solve the slowness of autoregressive speech synthesis is mainly to slice the whole sentence to achieve the purpose of parallelization. However, this method is essentially speech synthesis with an autoregressive structure, with a slow synthesis speed and does not fundamentally avoid the problem of low robustness. Non-autoregressive synthesis is fast and highly robust, but the synthesized speech quality and fluency are low.

[0003] From a practical perspective, this patent proposes a parallel speech synthesis method and apparatus, medium, and equipment based on prosodic features. First, the speech text to be synthesized is regularized; secondly, the regularized speech text is converted into a phoneme sequence and a phoneme prosodic sequence in parallel; then, the phoneme sequence and the phoneme prosodic sequence are predicted to have their Mel spectrograms using an acoustic model; finally, the Mel spectrogram is converted into a speech signal of the speech text to be synthesized using a HiFi-GAN vocoder. This patent combines the advantages of autoregressive speech synthesis and non-autoregressive speech synthesis, which can not only effectively improve the sound quality, fluency, speed, and robustness of speech synthesis, but also realize real-time and efficient speech synthesis. The system developed based on this patent can be widely used in intelligent speech synthesis in the field of artificial intelligence, such as intelligent customer service, intelligent audio, voice broadcasting, map navigation, audiobooks and other human-computer interaction scenarios. Summary of the invention

[0004] In view of the above problems, the present invention provides a parallel speech synthesis method, device, medium and equipment based on prosodic features.

[0005] According to one aspect of the present invention, there is provided a parallel speech synthesis method based on prosodic features, comprising:

[0006] (1) Regularizing the speech text to be synthesized, that is, converting the non-standardized speech text into standardized text according to a pre-set regularization rule;

[0007] (2) converting the regularized speech text into a phoneme sequence and a phoneme-level prosodic sequence in parallel;

[0008] (3) predicting the mel-spectrograms of the phoneme sequence and the phoneme-level prosody sequence using an acoustic model;

[0009] (4) Convert the Mel-spectrogram into a speech signal of the speech text to be synthesized using a vocoder.

[0010] Furthermore, the regularized speech text in step 2 is converted into a phoneme sequence, including:

[0011] The regularized speech text is converted into pinyin using a Chinese text transphoneme model G2PC based on conditional random fields;

[0012] The pinyin is mapped to corresponding phonemes using a custom pinyin and phoneme dictionary and the phoneme sequence is generated.

[0013] Furthermore, the regularized speech text in step 2 is converted into a phoneme-level prosodic sequence, including:

[0014] Performing word segmentation and part-of-speech tagging on the regularized speech text;

[0015] Using a prosody prediction network, converting the annotated regularized speech text into a word-level prosody sequence;

[0016] The word-level prosodic sequence is mapped to the phoneme-level prosodic sequence using a length alignment mechanism.

[0017] Furthermore, the rhythm prediction network is a language model or a pre-trained language model BERT consisting of two bidirectional long short-term memory artificial neural networks LSTM and two linear layers.

[0018] Furthermore, the acoustic model in step 3 includes an encoder module, a decoder module and a post-processing network;

[0019] The encoder module and the decoder module are both based on Conformer Blocks; the decoder module adopts a non-autoregressive structure;

[0020] The post-processing network corrects the predicted Mel-spectrogram and captures the long-term dependency and detail features between the phoneme sequence and the phoneme-level prosodic sequence to improve the sound quality and fluency of the synthesized speech.

[0021] Furthermore, the encoder module is a 4-layer Conformer Blocks, the decoder module is a 6-layer Conformer Blocks, the post-processing network is a 5-layer one-dimensional convolution, and the convolution kernel size is 1×3.

[0022] Furthermore, the vocoder in step 4 is a HiFi-GAN vocoder.

[0023] According to another aspect of the present invention, there is also provided a parallel speech synthesis device based on prosodic features, comprising:

[0024] An information acquisition module, adapted to acquire the speech text to be synthesized and regularize it;

[0025] A phoneme sequence generation module, adapted to convert the regularized speech text into a phoneme sequence;

[0026] A phoneme-level prosodic sequence generation module, adapted to convert the regularized speech text into a phoneme-level prosodic sequence;

[0027] A spectrum generation module, adapted to predict the mel-spectrogram of the phoneme sequence and the phoneme-level prosody sequence using an acoustic model;

[0028] The speech synthesis module is adapted to convert the Mel-spectrogram into a speech signal of the speech text to be synthesized by using a vocoder.

[0029] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute any of the above-mentioned parallel speech synthesis methods based on prosodic features.

[0030] According to another aspect of the present invention, a computing device is provided, wherein the computing device comprises a processor and a memory:

[0031] The memory is used to store program code and transmit the program code to the processor;

[0032] The processor is used to execute any of the above-mentioned parallel speech synthesis methods based on prosodic features according to the instructions in the program code.

[0033] Beneficial effects of the present invention:

[0034] This patent combines the advantages of autoregressive synthesis and non-autoregressive synthesis, which can not only effectively improve the sound quality, fluency, speed and robustness of speech synthesis, but also realize real-time and efficient speech synthesis. The system developed based on this patent can be widely used in the field of intelligent speech synthesis in the field of artificial intelligence, such as intelligent customer service, intelligent audio, voice broadcasting, map navigation, audiobooks and other human-computer interaction scenarios. The most intuitive judgment of the quality of the speech synthesis system is the quality of the synthesized audio, that is, the commonly used Mean Opinion Score (MOS) value. The MOS values ​​of the rhythm and fluency of the synthesized speech of this patent based on the Biaobei public speech synthesis data set are 4.16 and 4.05 respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of a parallel speech synthesis method based on prosodic features according to an embodiment of the present invention.

[0036] Figure 2 It is a structural diagram of a parallel speech synthesis method based on prosodic features according to an embodiment of the present invention.

[0037] Figure 3 It is a structural diagram of a text front-end processing module according to an embodiment of the present invention.

[0038] Figure 4 2 is a schematic diagram of a rhythm prediction structure according to an embodiment of the present invention.

[0039] Figure 5 Schematic diagram of the acoustic model structure of an embodiment of the present invention.

[0040] Figure 6 It is a schematic diagram of the structure of a parallel speech synthesis device based on prosodic features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will further describe in detail a prosodic feature-based parallel speech synthesis method, apparatus, medium, and device of the present invention in conjunction with specific embodiments.

[0042] Figure 1 FIG. 4 is a flow chart of a parallel speech synthesis method based on prosodic features according to an embodiment of the present invention. Figure 2 FIG. 4 shows a schematic diagram of the structure of a parallel speech synthesis method based on prosodic features according to an embodiment of the present invention. Figure 1 , Figure 2 It can be seen that the following steps are included:

[0043] S101: regularizing the speech text to be synthesized;

[0044] Input: speech text to be synthesized

[0045] Output: Regularized speech text

[0046] This module mainly converts non-standardized speech text into standardized text by manually setting regularization rules. Specific rules include but are not limited to:

[0047] Basic encoding conversion, for example: converting traditional characters to simplified characters.

[0048] Date regular expression, for example: convert 2022-03-28 to March 28, 2022.

[0049] Time regularization, for example: convert 10:30 to ten thirty minutes.

[0050] Temperature normalization, e.g. converting ° and ℃ to standard temperature forms degrees and Celsius.

[0051] Fraction regularization, for example: convert 1 / 2 to one-half.

[0052] Percent regular expressions, for example, convert 90% to ninety percent.

[0053] Regular expression of landline / mobile number and ID number, for example: convert 1 in landline / mobile number and ID number to 1.

[0054] Range regular expressions, for example, convert 1-9 to one to nine.

[0055] Number + quantifier regularization, for example: convert the quantifier 2 in 2 trees to two.

[0056] Number regular expression, convert Arabic numerals in phonetic text into Chinese characters.

[0057] URL and email regular expression, for example: convert www.sues.edu.cn to three parts: W.dotSUES.dotEDU.dotCN.

[0058] S102: Converting the regularized speech text into a phoneme sequence and a phoneme-level prosodic sequence in parallel;

[0059] Figure 3 FIG. 2 shows a schematic diagram of the structure of a text front-end processing module according to an embodiment of the present invention. Figure 3 As shown, the regularized speech text is converted into phoneme sequences and phoneme-level prosodic sequences respectively.

[0060] Regularized speech text converted into phoneme sequence

[0061] Input: Regularized speech text (output of S101)

[0062] Output: Phoneme sequence

[0063] First, the regularized speech text is converted into pinyin using the Chinese text to phoneme model G2PC (Grapheme-to-Phoneme for Chinese) based on conditional random fields, which can effectively handle the problem of Chinese polyphones, such as "hang" can be read as "hang" and "xing". Secondly, a custom pinyin and phoneme dictionary is used to map pinyin to corresponding phonemes and generate phoneme sequences.

[0064] Regularized speech text converted into phoneme-level prosodic sequences

[0065] Input: Regularized speech text (output of S101)

[0066] Output: Phoneme-level prosodic sequence

[0067] Figure 4 A schematic diagram of the rhythm prediction structure of an embodiment of the present invention is given. First, the regularized speech text is segmented and part-of-speech tagged. Secondly, two bidirectional long short-term memory artificial neural networks (Long Short-Term Memory, LSTM) and two linear layers are used to convert the tagged regularized speech text into a word-level rhythm sequence. Among them, the technology that can replace the rhythm prediction network based on two bidirectional LSTMs and two linear layers includes the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers).

[0068] Finally, the word-level prosodic sequence is mapped to the phoneme-level prosodic sequence using the length alignment mechanism. For example, if a word contains two phonemes, the prosodic sequence of this word is copied and expanded to the length of the phoneme.

[0069] S103: predicting the mel-spectrograms of the phoneme sequence and the phoneme-level prosody sequence using an acoustic model;

[0070] Input: Phoneme sequence (output of S102) and phoneme-level prosody sequence (output of S102)

[0071] Output: Mel-spectrogram

[0072] The acoustic model is the most important module for speech synthesis. Figure 5A schematic diagram of the acoustic model structure of an embodiment of the present invention is shown. This patent is based on the idea of ​​sequence to sequence (seq2seq), and uses traditional encoder modules and decoder modules in the acoustic model. In particular, the use of a non-autoregressive structure in the decoder makes the overall speech synthesis speed faster. At the same time, in order to make the speech synthesis sound quality and natural fluency higher, an acoustic model is built based on Conformer Blocks and the predicted Mel spectrum is corrected using a post-processing network, which can simultaneously capture the long-term dependencies and detail features between phoneme (rhythm) sequences, thereby improving the sound quality and fluency of the synthesized speech. In this patent, the encoder defaults to 4 layers of Conformer Blocks, the decoder defaults to 6 layers of Conformer Blocks, the post-processing network defaults to 5 layers of one-dimensional convolution, and the convolution kernel size is 1×3.

[0073] S104: Convert the mel-spectrogram into a speech signal using a vocoder.

[0074] Input: Mel-spectrogram (output of S103)

[0075] Output: Voice signal

[0076] The vocoder is also a key factor in synthesizing high-quality speech. Traditional methods mostly use statistical parameter models, such as Griffin-Lim. With the development of Generative Adversarial Net (GAN), GAN-based vocoder technology has developed rapidly, which not only has high speech synthesis quality, but also fast speech synthesis speed. This patent uses the HiFi-GAN vocoder (V1 version), and the alternative technologies include other vocoders such as Parallel Wave GAN (PWG).

[0077] The embodiment of the present invention also provides a parallel speech synthesis device based on prosodic features, such as Figure 6 As shown, including:

[0078] The information acquisition module 610 is adapted to acquire the speech text to be synthesized and regularize it;

[0079] A phoneme sequence generation module 620, adapted to convert the regularized speech text into a phoneme sequence;

[0080] A phoneme-level prosodic sequence generation module 630, adapted to convert the regularized speech text into a phoneme-level prosodic sequence;

[0081] A spectrum generation module 640, adapted to predict the mel-spectrograms of the phoneme sequence and the phoneme-level prosodic sequence using an acoustic model;

[0082] The speech synthesis module 650 is adapted to convert the Mel-spectrogram into a speech signal of the speech text to be synthesized by using a vocoder.

[0083] In an optional embodiment of the present invention, a computer-readable storage medium is further provided. The computer-readable storage medium is used to store program codes, and the program codes are used to execute the parallel speech synthesis method based on prosodic features of the above embodiment.

[0084] In an optional embodiment of the present invention, a computing device is also provided, which includes a processor and a memory: the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the parallel speech synthesis method based on prosodic features of the above-mentioned embodiment according to the instructions in the program code.

[0085] The embodiment of the present invention provides a parallel speech synthesis method and apparatus, medium, and equipment based on prosodic features. In the method provided by the embodiment of the present invention, the advantages of autoregressive speech synthesis and non-autoregressive speech synthesis are integrated, which can not only effectively improve the sound quality, fluency, speed, and robustness of speech synthesis, but also realize real-time and efficient speech synthesis. The system developed based on this patent can be widely used in the field of intelligent speech synthesis in the field of artificial intelligence, such as intelligent customer service, intelligent audio, voice broadcasting, map navigation, and audiobooks and other human-computer interaction scenarios. The most intuitive judgment of the quality of the speech synthesis system is the quality of the synthesized audio, that is, the commonly used MOS value. The MOS values ​​of the rhythm and fluency of the synthesized speech of this patent based on the public speech synthesis data set of Biaobei are 4.16 and 4.05 respectively.

[0086] The functional units in various embodiments of the present invention may be physically independent of each other, or two or more functional units may be integrated together, or all functional units may be integrated into one processing unit. The above integrated functional units may be implemented in the form of hardware, software or firmware.

[0087] Those skilled in the art can understand that if the integrated functional unit is implemented in the form of software and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can essentially or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, which includes a number of instructions to enable a computing device (such as a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention when running the instructions. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes.

[0088] Alternatively, all or part of the steps of implementing the aforementioned method embodiments may be accomplished by hardware associated with program instructions (such as a computing device such as a personal computer, a server, or a network device), and the program instructions may be stored in a computer-readable storage medium. When the program instructions are executed by a processor of a computing device, the computing device executes all or part of the steps of the methods described in the embodiments of the present invention.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate from the protection scope of the present invention.

Claims

1. A parallel speech synthesis method based on prosodic features, characterized in that: Includes steps: (1) Regularizing the speech text to be synthesized, that is, converting the non-standardized speech text into regularized speech text according to a pre-set regularization rule; (2) converting the regularized speech text into a phoneme sequence and a phoneme-level prosodic sequence in parallel; (3) predicting the mel-spectrograms of the phoneme sequence and the phoneme-level prosody sequence using an acoustic model; The acoustic model includes an encoder module, a decoder module and a post-processing network; The encoder module and the decoder module are both based on Conformer Blocks; the decoder module adopts a non-autoregressive structure; The post-processing network corrects the predicted mel-spectrogram and captures the long-term dependency and detail features between the phoneme sequence and the phoneme-level prosodic sequence to improve the sound quality and fluency of the synthesized speech; The encoder module is a 4-layer Conformer Blocks, the decoder module is a 6-layer Conformer Blocks, the post-processing network is a 5-layer one-dimensional convolution, and the convolution kernel size is 1×3; (4) Convert the Mel-spectrogram into a speech signal of the speech text to be synthesized using a vocoder.

2. A parallel speech synthesis method based on prosodic features according to claim 1, characterized in that: The step (2) of converting the regularized speech text into a phoneme sequence includes: The regularized speech text is converted into pinyin using a Chinese text transphoneme model G2PC based on conditional random fields; The pinyin is mapped to corresponding phonemes using a custom pinyin and phoneme dictionary and the phoneme sequence is generated.

3. The parallel speech synthesis method based on prosodic features according to claim 1, characterized in that: The step (2) of converting the regularized speech text into a phoneme-level prosodic sequence comprises: Performing word segmentation and part-of-speech tagging on the regularized speech text; Using a prosody prediction network, converting the annotated regularized speech text into a word-level prosody sequence; The word-level prosodic sequence is mapped to the phoneme-level prosodic sequence using a length alignment mechanism.

4. The parallel speech synthesis method based on prosodic features according to claim 3, characterized in that: The rhythm prediction network is a model composed of two bidirectional long short-term memory artificial neural networks LSTM and two linear layers or a pre-trained language model BERT.

5. The parallel speech synthesis method based on prosodic features according to claim 1, characterized in that: The vocoder in step (4) is a HiFi-GAN vocoder.

6. A parallel speech synthesis device based on prosodic features, characterized in that: include: The information acquisition module is adapted to acquire the speech text to be synthesized and regularize it to obtain the regularized speech text; A phoneme sequence generation module, adapted to convert the regularized speech text into a phoneme sequence; a phoneme level prosodic sequence generation module, adapted to convert the regularized speech text into a phoneme level prosodic sequence; A spectrum generation module, adapted to predict the mel-spectrogram of the phoneme sequence and the phoneme-level prosody sequence using an acoustic model; The acoustic model includes an encoder module, a decoder module and a post-processing network; The encoder module and the decoder module are both based on Conformer Blocks; the decoder module adopts a non-autoregressive structure; The post-processing network corrects the predicted mel-spectrogram and captures the long-term dependency and detail features between the phoneme sequence and the phoneme-level prosodic sequence to improve the sound quality and fluency of the synthesized speech; The encoder module is a 4-layer Conformer Blocks, the decoder module is a 6-layer Conformer Blocks, the post-processing network is a 5-layer one-dimensional convolution, and the convolution kernel size is 1×3; The speech synthesis module is adapted to convert the Mel-spectrogram into a speech signal of the speech text to be synthesized by using a vocoder.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the parallel speech synthesis method based on prosodic features according to any one of claims 1 to 5.

8. A computing device, characterized in that The computing device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the parallel speech synthesis method based on prosodic features according to any one of claims 1 to 5 according to the instructions in the program code.

Images