English speech synthesis method and system

Abstract

The present invention discloses an English speech synthesis method and system, the method comprising: performing digital processing, abbreviation processing and proper noun processing on the input English text respectively; applying the English letters and / or phonemes in the processed English text to an acoustic model to obtain the spectral features corresponding to the English text, wherein the acoustic model includes an Embedding module of the same dimension as the English text input type; the spectral features are input into a vocoder, and the spectral features are mapped into speech. The present invention performs front-end processing on the English text input by the user, thereby improving the accuracy of subsequent synthesis, and selects different combinations of acoustic models and vocoders for different end-to-end and mobile application scenarios, which can meet the corresponding needs and improve the accuracy, naturalness and synthesis efficiency of English speech synthesis.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis, and in particular to an English speech synthesis method and system. Background Art

[0002] Speech synthesis is the task of converting input text into standard fluent human speech. The quality evaluation of speech synthesis systems is generally divided into the following aspects: naturalness, robustness, and accuracy of generated speech.

[0003] Traditional speech synthesis technologies include concatenation and statistical parameter methods. Among them, the concatenation method records a large amount of speech in advance, and selects a large number of basic units according to the text content to connect them into fluent speech during synthesis. However, this method has a high coverage requirement for the recorded speech, and the database established is also relatively large. When it is actually used on the device, it is often limited by the capacity of the device; the statistical parameter method is to establish a mapping from text features to acoustic features based on a statistical model, and then use these acoustic features to restore the waveform. This method does not require pre-recorded speech, but the quality of the synthesized speech is low. These traditional methods are all based on the analysis and synthesis of acoustic features. These methods are widely used in the industry because of their strong robustness and accuracy. However, the common disadvantage of these methods is that the synthesized speech has great defects in naturalness.

[0004] The existing English speech synthesis model has the following problems: first, there is a lack of front-end processing of English text, which results in some content in the text being unable to be pronounced correctly; second, the acoustic model part still uses traditional synthesis technology, and the low naturalness and complex synthesis process no longer meet existing needs; finally, the vocoder part still uses traditional vocoders or models with low synthesis efficiency, which are also difficult to meet the needs in terms of quality and performance. Summary of the invention

[0005] Therefore, in order to overcome the defects that the existing English speech synthesis model cannot accurately complete the correct pronunciation corresponding to the English text and has low efficiency, the present invention provides an English speech synthesis method and system.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides an English speech synthesis method, comprising:

[0008] The input English text is processed into numbers, abbreviations and proper nouns respectively;

[0009] The English letters and / or phonemes in the processed English text are put into the acoustic model to obtain the spectral features corresponding to the English text, wherein the acoustic model includes an Embedding module of the same dimension as the English text input type;

[0010] The spectral features are input into a vocoder, and the spectral features are mapped into speech.

[0011] Preferably, the process of digital processing of English text includes: dividing the digital reading of English into digital reading, numerical reading and special reading, setting corresponding rule tables for digital reading and numerical reading for matching, if a word representing a unit is matched after the number, the number string is processed according to the numerical reading; if a word representing a content attribute is matched before the number or the number appears alone, the number string is processed according to the numerical reading; setting a preset mapping table for the special reading, and when matching preset special expressions, including: date, radio station, time, finding the special reading corresponding to the special expression by looking up the table.

[0012] Preferably, the process of abbreviating the English text includes: establishing a mapping table to match abbreviations with corresponding full spellings, and when it is detected that the acquired text contains abbreviations, the abbreviations are replaced by the corresponding full spellings obtained by looking up the table.

[0013] Preferably, the process of processing proper nouns on English text includes: establishing a preset proper noun mapping table to map the proper nouns to phonemes of corresponding pronunciations.

[0014] Preferably, the process of training the acoustic model includes:

[0015] The English data set used for training only contains English words but no phonemes. The words in each sentence of the input English text are traversed and randomly replaced with corresponding phonemes with a preset probability. The Embedding module converts the sequence of letters and phonemes in the data into corresponding vectors for subsequent operations to train the sequence-to-sequence generation model with an attention mechanism. The trained English acoustic model is capable of processing three input methods at the same time.

[0016] Preferably, the acoustic model adopts Tacotron 2, a sequence-to-sequence generation model with an attention mechanism, and the vocoder adopts WaveGlow, a stream-based generation model.

[0017] Preferably, the acoustic model adopts Tacotron 2, a sequence-to-sequence generation model with an attention mechanism that is compressed according to a preset compression ratio, and the vocoder adopts Squeeze_Wave, a stream-based generation model.

[0018] Preferably, the Tacotron2 includes an encoder, a decoder and an attention module, wherein the encoder includes: 1 embedding layer, 3 convolutional layers, and 1 bidirectional long short-term memory layer; the decoder includes: 2 fully connected layers, 2 unidirectional long short-term memory layers, 1 mapping layer and 5 convolutional layers. The number of filters in the convolutional layers of the encoder and decoder and the number of hidden units in the LSTM and fully connected layers are compressed to 50% of the original as the network structure of the acoustic model.

[0019] In a second aspect, an embodiment of the present invention provides an English speech synthesis system, comprising:

[0020] The English front-end processing module is used to process the input English text into numbers, abbreviations and proper nouns respectively;

[0021] An acoustic model module, used to put the English letters and / or phonemes in the processed English text into the acoustic model to obtain the spectral features corresponding to the English text, wherein the acoustic model includes an Embedding module of the same dimension as the English text input type;

[0022] The vocoder module is used to input the spectrum feature into a vocoder and map the spectrum feature into speech.

[0023] In a third aspect, an embodiment of the present invention provides a computer device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the English speech synthesis method described in the first aspect of the embodiment of the present invention.

[0024] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the English speech synthesis method described in the first aspect. .

[0025] The technical solution of the present invention has the following advantages:

[0026] The English speech synthesis method and system provided by the embodiment of the present invention respectively processes the input English text into numbers, abbreviations and proper nouns; the English letters and / or phonemes in the processed English text are input into the acoustic model to obtain the spectral features corresponding to the English text, and the acoustic model includes an Embedding module of the same dimension as the English text input type; the spectral features are input into the vocoder to map the spectral features into speech. The embodiment of the present invention performs front-end processing on the English text input by the user, thereby improving the accuracy of subsequent synthesis. For different end-to-end and mobile application scenarios, different combinations of acoustic models and vocoders are selected to meet the corresponding needs and improve the accuracy, naturalness and synthesis efficiency of English speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A flowchart of an example of an English speech synthesis method provided in an embodiment of the present invention;

[0029] Figure 2 A module composition diagram of an example of an English speech synthesis system provided in an embodiment of the present invention;

[0030] Figure 3 A composition diagram of a specific example of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0033] Example 1

[0034] The embodiment of the present invention provides an English speech synthesis method, which can be applied to end-to-end and mobile English speech synthesis application scenarios, such as Figure 1 As shown, including:

[0035] Step S1: Perform digital processing, abbreviation processing and proper noun processing on the input English text.

[0036] In different contexts, the same Arabic numerals can represent years, telephone numbers, currency denominations, etc. For example, 2020 is pronounced as "twenty-twenty" when used as a year, and as "two-thousand-and-twenty" when used as a currency. If not processed, the incorrect English reading will make it difficult for the listener to understand. In the embodiment of the present invention, in order to correctly use English to represent Arabic numerals, the process of digital processing of English text includes:

[0037] The English digit reading is divided into digital reading (i.e., reading one digit at a time, such as a telephone number, etc.), numerical reading (i.e., reading a string of digits as a whole, such as a statistical number, etc.) and special reading (such as year, time, date, radio station, etc.), and corresponding rule tables are set for digital reading and numerical reading for matching. If a word representing a unit is matched after a digit, the digit string is processed according to the numerical reading, such as if a word representing a unit such as meter, kilogram, etc. is matched after a digit, the digit string is processed according to the numerical reading; if a word representing a content attribute is matched before a digit, the digit string is processed according to the numerical reading, such as if a word such as Tel is matched before and after a digit or the digit only appears alone; a preset mapping table is set for the special reading, and when a preset special expression is matched, the special reading corresponding to the special expression is found by looking up the table.

[0038] For user input of abbreviations, such as "dr" is pronounced as "doctor", "jr" is pronounced as "junior", etc., if it is not processed by the front end, the correct voice cannot be generated. By processing the abbreviations, the abbreviations can be correctly expressed in English. The process of abbreviation processing of English text in the embodiment of the present invention includes: establishing a mapping table to match the abbreviations with the corresponding full spellings, and when it is detected that the acquired text contains abbreviations, the abbreviations are replaced by the corresponding full spellings obtained by looking up the table.

[0039] In English texts, proper nouns are often present. By processing them, the pronunciation of proper nouns can be correctly expressed in English. The process of processing proper nouns in the embodiment of the present invention includes: establishing a preset proper noun mapping table, mapping the proper nouns to phonemes of corresponding pronunciations. For example, "FBI" is mapped to "EH FB IY AY", etc. The model has the ability to use phonemes as input and English letters as input at the same time, so the probability of errors in synthesizing speech by the model can be reduced.

[0040] The front-end processing is performed through the English text entered by the user, which improves the accuracy of subsequent results.

[0041] Step S2: The English letters and / or phonemes in the processed English text are put into the acoustic model to obtain the spectral features corresponding to the English text, and the acoustic model includes an Embedding module of the same dimension as the English text input type.

[0042] The acoustic model encoder in the embodiment of the present invention includes an Embedding module, etc. In the English speech synthesis method proposed in the embodiment of the present invention, the input text is mainly composed of English letters and / or phonemes. Among them, English letters include 26 letters, and phonemes include all phonemes appearing in CMU DICT. According to the types of these inputs in the text, an Embedding module of the same dimension is designed, so that the corresponding word vectors are used in the model to represent different characters.

[0043] In order to enable the acoustic model to process three input modes for synthesis at the same time, the embodiment of the present invention proposes a new English acoustic model training strategy, and the training process includes:

[0044] The English data set used for training only contains English words but no phonemes. The words in each sentence of the input English text are traversed, and the words in the sentence are randomly replaced with corresponding phonemes with a preset probability (for example, 50%) (the mapping table is CMU DICT. If the word is a polyphone, it is not replaced to ensure the reliability of the training data). The Embedding module converts the sequence composed of letters and phonemes in the data into corresponding vectors for subsequent operations to train the sequence-to-sequence generation model with an attention mechanism. The trained English acoustic model has the ability to process three input methods at the same time.

[0045] Step S3: input the spectrum features into a vocoder to map the spectrum features into speech.

[0046] The embodiment of the present invention adopts a stream-based generation model, which can map spectrum features into high-quality speech.

[0047] In a specific embodiment, if it is for end-to-end English speech synthesis, the acoustic model uses Tacotron 2, a sequence-to-sequence generative model with an attention mechanism, and the vocoder uses WaveGlow, a stream-based generative model. Tacotron 2 is a sequence-to-sequence generative model with an attention mechanism, which realizes the modeling from text to spectral features. A large number of experiments have proved that this model can learn the alignment between text and spectrum, and generate natural and rhythmic speech, which meets the requirements of modern speech synthesis for naturalness. WaveGlow is a stream-based generative model that can map spectral features to high-quality speech. Compared with neural network vocoders such as WaveNet, WaveGlow has the characteristics of non-autoregression, so it has higher inference efficiency and can meet the high concurrency requirements in actual production environments.

[0048] In a specific embodiment, if it is for English speech synthesis on a mobile terminal, due to the limited resources on the mobile terminal, the model needs to be compressed. In order to use the tacotron2 module on the mobile terminal device, the embodiment of the present invention cuts the number of channels of the model to varying degrees, so that the amount of calculation and the number of parameters of the model are greatly reduced. In order to improve the efficiency of autoregressive reasoning, a strategy of decoding multiple frames each time is adopted to multiply the reasoning speed on the mobile terminal with almost no loss of sound quality. Thanks to the above two strategies, the Tacotron2 model can perform low-latency reasoning on mobile devices.

[0049] The Tacotron2 model includes: encoder, decoder and attention module. The encoder includes: 1 embedding layer, 3 convolution layers, 1 bidirectional long short-term memory layer; the decoder includes: 2 fully connected layers, 2 unidirectional long short-term memory layers, 1 mapping layer and 5 convolution layers. In the actual test process, the number of convolution layer filters of the encoder and decoder parts was reduced to 75%, 50%, and 25% respectively, and the number of hidden units of the LSTM and fully connected layers of the encoder and decoder parts was reduced to 75%, 50%, and 25%. Finally, the experiment concluded that the effect and performance obtained by cutting all to 50% are the most suitable parameters for mobile terminals.

[0050] 100％ 75％ 50％ 25％ Encoder CNN 512 384 256 128 Encoder LSTM 512 384 256 128 Decoder FC 256 192 128 64 Decoder LSTM 1024 786 512 256 Decoder CNN 512 384 256 128

[0051] Therefore, the number of filters in the convolutional layers of the encoder and decoder and the number of hidden units in the LSTM and fully connected layers are finally compressed to 50% of the original as the network structure of the acoustic model.

[0052] The mobile vocoder uses Squeeze_wave, which is a stream-based generative model that can map spectral features to high-quality speech. Squeeze_wave analyzes the model parameters and computational complexity of WaveGlow and adopts more efficient computing units. Therefore, it greatly reduces the number of model parameters and computational complexity without sacrificing sound quality, making it highly real-time on mobile devices. Compared with neural network vocoders such as WaveNet, Squeeze_wave has the characteristics of non-autoregression, which enables it to have higher inference efficiency and can meet the real-time rate requirements of mobile devices.

[0053] The English speech synthesis method provided in the embodiment of the present invention performs front-end processing on the English text input by the user, thereby improving the accuracy of subsequent synthesis. Different combinations of acoustic models and vocoders are selected for different end-to-end and mobile application scenarios, which can meet corresponding needs and improve the accuracy, naturalness and synthesis efficiency of English speech synthesis.

[0054] Example 2

[0055] The embodiment of the present invention provides an English speech synthesis system, such as Figure 2 As shown, including:

[0056] The English front-end processing module 1 is used to process the input English text into numbers, abbreviations and proper nouns respectively; after being processed by the English front-end processing module, the number of erroneous speech generated by the subsequent modules will be greatly reduced. This module executes the method described in step 1 of embodiment 1, which will not be repeated here.

[0057] Acoustic model module 2 is used to put the English letters and / or phonemes in the processed English text into the acoustic model to obtain the spectral features corresponding to the English text, and the acoustic model includes an Embedding module of the same dimension as the English text input type; this module executes the method described in step S2 of Example 1, which will not be repeated here.

[0058] The vocoder module 3 is used to input the spectrum feature into the vocoder and map the spectrum feature into speech. This module executes the method described in step S3 of embodiment 1, which will not be described in detail here.

[0059] The English speech synthesis system provided by the embodiment of the present invention improves the accuracy of synthesis by setting an English front-end processing module, improves the naturalness of synthesis by an acoustic model module, and improves the efficiency of synthesis by a vocoder module. Compared with the existing system, the system provided by the embodiment of the present invention has the advantages of high accuracy, high naturalness, and high synthesis efficiency, and can meet the use requirements in different scenarios.

[0060] Example 3

[0061] An embodiment of the present invention provides a computer device, such as Figure 3 As shown, it includes: at least one processor 401, such as a CPU (Central Processing Unit), at least one communication interface 403, a memory 404, and at least one communication bus 402. Among them, the communication bus 402 is used to realize the connection and communication between these components. Among them, the communication interface 403 may include a display screen (Display), a keyboard (Keyboard), and the optional communication interface 403 may also include a standard wired interface and a wireless interface. The memory 404 may be a high-speed RAM memory (Ramdom Access Memory, volatile random access memory) or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 404 may also be at least one storage device located away from the aforementioned processor 401. Among them, the processor 401 can execute the method described in Example 1. A set of program codes are stored in the memory 404, and the processor 401 calls the program code stored in the memory 404 to execute the English speech synthesis method of Example 1.

[0062] The communication bus 402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The fact that only one line is used in the diagram does not mean that there is only one bus or only one type of bus.

[0063] The memory 404 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 404 may also include a combination of the above-mentioned types of memory. The processor 401 may be a central processing unit (English: central processing unit, abbreviated: CPU), a network processor (English: network processor, abbreviated: NP) or a combination of CPU and NP.

[0064] Wherein, the processor 401 may further include a hardware chip. The above-mentioned hardware chip may be an application-specific integrated circuit (English: application-specific integrated circuit, abbreviated: ASIC), a programmable logic device (English: programmable logic device, abbreviated: PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (English: complex programmable logic device, abbreviated: CPLD), a field programmable gate array (English: field-programmable gate array, abbreviated: FPGA), a generic array logic (English: generic array logic, abbreviated: GAL) or any combination thereof. Optionally, the memory 404 is also used to store program instructions. The processor 401 may call program instructions to implement the English speech synthesis method as in the embodiment 1 of the present application.

[0065] The embodiment of the present invention further provides a computer-readable storage medium, on which computer-executable instructions are stored, and the computer-executable instructions can execute the English speech synthesis method of embodiment 1. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0066] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the scope of protection of the invention.

Claims

1. An English speech synthesis method, characterized in that: include: The input English text is processed into numbers, abbreviations and proper nouns respectively. The process of processing the English text into proper nouns includes: establishing a preset proper noun mapping table to map the proper nouns into phonemes of corresponding pronunciations; Inputting English letters and / or phonemes in the processed English text into an acoustic model to obtain spectral features corresponding to the English text, wherein the acoustic model includes an Embedding module of the same dimension as the English text input type; the process of training the acoustic model includes: The English dataset used for training only contains English words but no phonemes. The words in each sentence of the input English text are traversed and randomly replaced with corresponding phonemes with a preset probability. The Embedding module converts the sequence of letters and phonemes in the data into corresponding vectors for subsequent operations to train the sequence-to-sequence generation model with attention mechanism. The trained English acoustic model has the ability to process three input methods at the same time. The spectral features are input into a vocoder, and the spectral features are mapped into speech.

2. The English speech synthesis method according to claim 1, characterized in that: The process of digital processing of English text includes: The English numeral reading is divided into numeral reading, numerical reading and special reading. Corresponding rule tables are set for numeral reading and numerical reading for matching. If a word representing a unit is matched after a numeral, the numeral string is processed according to the numerical reading; if a word representing a content attribute is matched before a numeral or a numeral appears alone, the numeral string is processed according to the numerical reading; a preset mapping table is set for the special reading. When matching preset special expressions, including date, radio station and time, the special reading corresponding to the special expression is found by looking up the table.

3. The English speech synthesis method according to claim 1, characterized in that: The process of abbreviating English text includes: establishing a mapping table to match abbreviations with corresponding full spellings, and when it is detected that the acquired text contains abbreviations, the abbreviations are replaced by the corresponding full spellings obtained by looking up the table.

4. The English speech synthesis method according to claim 1, characterized in that: The acoustic model adopts Tacotron 2, a sequence-to-sequence generation model with an attention mechanism, and the vocoder adopts WaveGlow, a stream-based generation model.

5. The English speech synthesis method according to claim 1, characterized in that: The acoustic model adopts Tacotron 2, a sequence-to-sequence generation model with an attention mechanism that is compressed according to a preset compression ratio, and the vocoder adopts Squeeze_Wave, a stream-based generation model.

6. The English speech synthesis method according to claim 5, characterized in that: The Tacotron2 includes an encoder, a decoder and an attention module, wherein the encoder includes: 1 embedding layer, 3 convolutional layers, and 1 bidirectional long short-term memory layer; the decoder includes: 2 fully connected layers, 2 unidirectional long short-term memory layers, 1 mapping layer and 5 convolutional layers. The number of filters in the convolutional layers of the encoder and decoder and the number of hidden units in the LSTM and fully connected layers are compressed to 50% of the original as the network structure of the acoustic model.

7. An English speech synthesis system, characterized in that: include: The English front-end processing module is used to process the input English text into numbers, abbreviations and proper nouns respectively. The process of processing the English text into proper nouns includes: establishing a preset proper noun mapping table to map the proper nouns into phonemes of corresponding pronunciations; The acoustic model module is used to input the English letters and / or phonemes in the processed English text into the acoustic model to obtain the spectral features corresponding to the English text, and the acoustic model includes an Embedding module of the same dimension as the English text input type; the process of training the acoustic model includes: The English dataset used for training only contains English words but no phonemes. The words in each sentence of the input English text are traversed and randomly replaced with corresponding phonemes with a preset probability. The Embedding module converts the sequence of letters and phonemes in the data into corresponding vectors for subsequent operations to train the sequence-to-sequence generation model with attention mechanism. The trained English acoustic model has the ability to process three input methods at the same time. The vocoder module is used to input the spectrum feature into a vocoder and map the spectrum feature into speech.

8. A computer device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the English speech synthesis method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the English speech synthesis method according to any one of claims 1 to 6.

Images