Parallel speech synthesis method, device, equipment and computer-readable storage medium

Abstract

The present disclosure provides a parallel speech synthesis method, apparatus, device and computer-readable storage medium. The method includes splitting a text into multiple segments, and then obtaining multiple initial hidden states for a recurrent neural network of the multiple segments based on the text. The method also includes synthesizing the multiple segments in parallel based on the multiple initial hidden states and the input features of the multiple segments. In the embodiment of the present disclosure, in the process of synthesizing multiple segments in parallel using a recurrent neural network, an initial hidden state is provided for each segment through a hidden state prediction model, which can not only improve the speed of speech synthesis and realize real-time speech synthesis, but also alleviate the interruption of hidden states between segments, thereby ensuring the quality of synthesized speech.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of speech synthesis technology, and more specifically to a method, apparatus, device, computer-readable storage medium, and computer program product for synthesizing speech in parallel using a recurrent neural network (RNN). Background Art

[0002] Speech synthesis refers to the technology of converting text into speech, also known as Text-to-Speech (TTS). Generally, speech synthesis technology converts text information into speech information with good sound quality and high natural fluency through computers. Speech synthesis is one of the core technologies of intelligent voice interaction technology, and together with speech recognition technology, it constitutes an indispensable part of intelligent voice interaction.

[0003] Traditional speech synthesis mainly includes speech synthesis methods based on vocoder parameters and speech synthesis methods based on unit selection and splicing. Generally speaking, the quality of speech synthesis (including sound quality and natural fluency) directly affects the user's hearing and the user experience of related products. In recent years, with the development of deep learning technology and its wide application in the field of speech synthesis, the sound quality and natural fluency of speech synthesis have been significantly improved. In addition, with the rapid popularization of intelligent hardware, the scenarios for using speech synthesis to obtain information are becoming more and more abundant. At present, speech synthesis has been widely used in voice broadcasting, map navigation, intelligent customer service, smart speakers and other fields and products. Summary of the invention

[0004] According to an exemplary embodiment of the present disclosure, a parallel speech synthesis method, apparatus, device, and computer-readable storage medium are provided.

[0005] In a first aspect of the present disclosure, a parallel speech synthesis method is provided. The method comprises: splitting a text into multiple segments; obtaining multiple initial hidden states for a recurrent neural network of the multiple segments based on the text; and synthesizing the multiple segments in parallel based on the multiple initial hidden states and input features of the multiple segments.

[0006] In a second aspect of the present disclosure, a parallel speech synthesis device is provided. The device includes: a segment splitting module configured to split a text into multiple segments; a hidden state acquisition module configured to obtain multiple initial hidden states for a recurrent neural network of multiple segments based on a text; and a parallel speech synthesis module configured to synthesize multiple segments in parallel based on the multiple initial hidden states and input features of the multiple segments.

[0007] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements a method or process according to an embodiment of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method or process according to the embodiment of the present disclosure is implemented.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, which comprises a computer program, which implements the method according to the first aspect of the present disclosure when being executed by a processor.

[0010] It should be understood that the contents described in the summary of the present invention are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram showing an example parallel speech synthesis scenario according to an embodiment of the present disclosure is shown;

[0013] Figure 2 A flow chart of a parallel speech synthesis method according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram showing a process of performing real-time speech synthesis of multiple segments in parallel based on the continuity of hidden states of the segments according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A schematic diagram showing a process of serially synthesizing each segment in an autoregressive manner according to an embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram showing an example architecture of a RNN-based speech synthesis system according to an embodiment of the present disclosure is shown;

[0017] Figure 6 A schematic diagram showing a training process of a speech synthesis system based on an RNN according to an embodiment of the present disclosure is shown;

[0018] Figure 7A block diagram showing a parallel speech synthesis device according to an embodiment of the present disclosure; and

[0019] Figure 8 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0022] Traditional speech synthesis systems are mainly divided into two types: parameter systems based on vocoders and waveform splicing systems based on unit selection. The parameter system based on vocoders first maps the text input representation into acoustic parameters such as spectrum and fundamental frequency, and then uses the vocoder to convert these features into speech. The waveform splicing system based on unit selection also maps the text input representation into acoustic parameters such as spectrum and fundamental frequency, and at the same time uses text rules and unit selection strategies such as acoustic target cost and connection cost to select the best waveform segment sequence from the speech library, and finally splices the selected segments to synthesize the target speech. The parameter system based on the vocoder has a relatively high fluency because it uses an acoustic model to predict acoustic parameters; however, the vocoder is a simplified algorithm based on the sound source-vocal tract model according to the human pronunciation mechanism, resulting in low synthesis quality of the parameter system. The waveform splicing system based on unit selection directly selects the original sound segment from the speech library, so it can ensure high sound quality; however, once the speech segment is not well selected, it will lead to the problem of discontinuous splicing, and the natural fluency is often not high. It can be seen from this that it is difficult for traditional speech synthesis systems to strike a balance between sound quality and natural fluency. The quality of synthesized speech is quite different from that of natural speech, and its quality is usually lower.

[0023] In recent years, the improvement of the traditional speech synthesis system is the neural speech synthesis (Neural TTS) system using deep learning technology. It can use a learnable deep model to directly model the speech sampling points, avoiding the complex design of the traditional synthesis system, and the synthesized sound quality and natural fluency have been greatly improved. The speech synthesized by the neural speech synthesis technology has not only good sound quality but also high fluency. However, neural speech synthesis generally uses a stacked multi-layer network structure or a complex structure to model the speech sampling points, which results in a large amount of calculation required for each step of speech sampling point generation. Therefore, neural speech synthesis has a high computational cost. Taking the RNN-based speech synthesis system as an example, RNN generates speech in a serial manner. For example, to generate a speech with a sampling frequency of 16,000 per second, 16,000 forward calculations need to be performed in sequence. The normal calculation time required will be much more than 1 second. This high delay leads to an extremely low real-time rate. Therefore, although the RNN-based speech synthesis system has a high speech synthesis quality, it is difficult to meet the requirements of real-time speech synthesis due to its large amount of calculation and single-point recursive properties.

[0024] In order to achieve real-time speech synthesis based on RNN, the main improvement methods include the following three. First, reduce the amount of calculation of single-step operation. The most direct way is to reduce the dimension of the hidden layer, but this will directly lead to performance loss and the quality of synthesized sound will be significantly reduced; another way is to reduce the number of non-zero weights by sparse weight matrix, which can maintain the dimension of the hidden layer unchanged and the representation ability of the hidden layer. In addition, nonlinear functions with lower computational complexity (such as softsign function) can be used to replace the sigmoid or tanh nonlinear functions of the original gated recurrent unit (GRU). However, the above-mentioned simplified processing to reduce the amount of single-step calculation will bring performance loss. Second, perform graphics processing unit (GPU) kernel optimization. Ordinary GPU implementation cannot directly achieve fast real-time synthesis. The main bottleneck is the bandwidth limitation of communication between video memory and registers and the overhead caused by initiating each kernel operation. In order to improve the computing efficiency of the GPU, on the one hand, the number of times the register copies data from the video memory can be reduced, and the model parameters can be read into the register at one time. One of the limitations is that the number of registers needs to match the number of model parameters; on the other hand, the number of kernel initiations can be reduced. Once all model parameters can be read into the register, the generation of sampling points for the entire sentence can be optimized and merged into one kernel operation, thereby avoiding the overhead caused by a large number of kernel initiations. In addition, it requires a GPU with a high-performance computing architecture to support real-time computing, and the hardware cost is high. Third, subscale batch sampling point generation. The subscale strategy supports the parallel generation of multiple sampling points by decomposing and simplifying the probability of the sampling point sequence, but this destroys the timing dependency of the sampling points, causing the interruption problem of the RNN hidden state, which will bring performance loss. In addition, the subscale has the defect of hard delay of the first packet. In the scenario where the first packet has high real-time performance, the subscale will bring a large delay. It can be seen that although the above three improvement methods can accelerate speech synthesis by simplifying the single-step calculation amount of the model, optimizing and accelerating with high-performance hardware, and generating subscale batch sampling points, they will sacrifice sound quality, resulting in poor synthesized speech quality.

[0025] The inventors of the present disclosure noticed that RNN has natural timing dependencies (such as hidden state connections), which determines that RNN is difficult to execute in parallel in theory and can only be generated step by step. In order to achieve real-time speech synthesis based on RNN, the embodiments of the present disclosure propose a parallel speech synthesis scheme based on the continuity of hidden states of fragments. In the process of using RNN to synthesize multiple fragments in parallel, the embodiments of the present disclosure provide an initial hidden state for each fragment through a hidden state prediction model, which can not only improve the speed of speech synthesis and achieve real-time speech synthesis, but also alleviate the interruption of hidden states between fragments, thereby ensuring the quality of synthesized speech by ensuring the continuity of hidden states within the RNN.

[0026] The RNN parallel real-time speech synthesis technology based on the hidden state continuity of the fragments proposed in the present disclosure creatively solves the problem of using RNN for online real-time synthesis, and significantly improves the speed of RNN synthesis. The scheme of the present disclosure can not only ensure that the synthesized speech has a high quality, but also support large-scale online deployment. In some embodiments, the parallel RNN synthesis technology proposed in the present disclosure uses fragments (such as phonemes, syllables, words, etc.) as basic synthesis units, multiple fragments are synthesized in parallel, and each fragment is serially synthesized in an autoregressive manner; at the same time, in order to ensure the continuity of the RNN hidden state between fragments, the present disclosure proposes to use a hidden state prediction network to provide an initial hidden state for each fragment, which effectively solves the RNN hidden state interruption problem caused by parallel synthesis and ensures the high quality of parallel synthesis. This RNN parallel real-time speech synthesis technology based on the hidden state continuity of fragments has cleared the biggest obstacle to real-time synthesis using RNN, and has greatly promoted the transformation of speech synthesis technology from traditional parameter systems and splicing systems to neural speech synthesis systems.

[0027] Figure 1 FIG. 1 is a schematic diagram of an example parallel speech synthesis scenario 100 according to an embodiment of the present disclosure. It should be understood that the scenario 100 is only an example scenario that can be implemented by the embodiment of the present disclosure, and is not intended to limit the scope of protection of the present disclosure. Figure 1 As shown, for the input text 110 to be synthesized speech (e.g., text 115 "Tomorrow the weather will be fine, with a northerly wind of level 3 to 4"), the text is first analyzed at box 120. For example, the text can be converted into phonetic characters to determine the pronunciation of each character, and in the case of polyphonic characters, the pronunciation of polyphonic characters can be predicted. In addition, the text can be analyzed for prosody to mark prosody information such as stress and pauses.

[0028] Next, speech synthesis is performed at box 130. In an embodiment of the present disclosure, a speech synthesis process is performed using an RNN-based speech synthesis model, such as a WaveRNN model. It should be understood that any known or future developed RNN-based speech synthesis model can be used in conjunction with the embodiments of the present disclosure. In an embodiment of the present disclosure, since the initial hidden state of the RNN for each segment can be predicted and obtained, multiple segments can be synthesized in parallel with almost no impact on the speech quality. In the context of the present disclosure, the term "initial hidden state" may refer to the initial hidden state in the RNN when synthesizing each segment. Figure 1 As shown, the embodiment of the present disclosure can simultaneously synthesize segment 1 and segment 2, etc., so as to obtain output speech 140, such as speech 145. Figure 2-8 An example implementation of parallel synthesized speech is described.

[0029] It should be understood that the parallel speech synthesis method according to the embodiment of the present disclosure can be deployed in various electronic devices. For example, in the scenario of the client-server architecture, the parallel synthesis speech method according to the embodiment of the present disclosure can be implemented on the client side or on the server side. Alternatively, the parallel synthesis speech method according to the embodiment of the present disclosure can also be partially implemented on the client side and the other part on the server side.

[0030] Figure 2 FIG. 2 is a flow chart of a parallel speech synthesis method 200 according to an embodiment of the present disclosure. Figure 3 The method 200 is described together with a parallel speech synthesis process 300 .

[0031] In block 202, a text is split into multiple segments. Figure 3 , for the text 305 of the speech to be synthesized, the text is split into multiple segments, such as segments 311, 312, and 313. In some embodiments, each segment can be any one of a phoneme, a syllable, and a prosodic word, or even a larger pronunciation unit, wherein a phoneme is the smallest unit constituting a syllable, which is the smallest type of speech segment, and phonemes include two major categories: vowels and consonants. A syllable is a basic unit of pronunciation, which may include one or more phonemes. For example, in Chinese, a Chinese character may be a syllable. A prosodic word refers to a word defined from the perspective of prosody, which may include multiple syllables. It should be understood that the segments in the embodiments of the present disclosure may also be larger pronunciation units. For example, when each segment is a syllable, the text (e.g., Chinese text) is split according to each Chinese character, and a syllable (corresponding to a Chinese character) is a segment.

[0032] In block 204, based on a piece of text, multiple initial hidden states for a recurrent neural network are obtained for multiple segments. Figure 3 According to the hidden state prediction model 320 of the embodiment of the present disclosure, the initial hidden state of the RNN of each segment can be predicted for subsequent parallel speech synthesis. RNN has a natural timing dependency, and the calculation at the next moment usually requires the hidden state generated at the previous moment. If the speech is synthesized in parallel, the traditional method will cause the interruption of the hidden state. In contrast, the embodiment of the present disclosure uses the pre-trained hidden state prediction model 320 to predict the initial hidden state of each segment in advance, without having to wait for the speech synthesis at the previous moment to be completed before performing the subsequent speech synthesis. In this way, the continuity of the hidden state can be guaranteed.

[0033] In block 206, multiple segments are synthesized in parallel based on the multiple initial hidden states and the input features of the multiple segments. Figure 3As shown, the RNN-based speech synthesis model 330 can synthesize multiple segments at the same time according to the initial hidden state of each segment, without waiting for the previous segment to be synthesized before synthesizing the next segment. Therefore, the embodiment of the present disclosure provides an initial hidden state for each segment through a hidden state prediction model, which can not only improve the speed of speech synthesis and realize real-time speech synthesis, but also alleviate the hidden state interruption between segments, thereby ensuring the quality of synthesized speech.

[0034] Therefore, the embodiment of the present disclosure proposes a RNN parallel real-time speech synthesis technology based on the hidden state continuity of the fragment. This technology uses the fragment of speech as the basic synthesis unit of RNN. From the perspective of phonetics, the fragment can include phonemes, syllables, rhythmic words, and even larger pronunciation units. A given text to be synthesized can be divided into multiple fragments, and then the multiple fragments are synthesized in parallel, and each fragment can be synthesized serially in an autoregressive manner. This method of parallel synthesis according to the fragments significantly improves the speed of RNN synthesis and meets the requirements of real-time synthesis. Due to the internal timing dependence, RNN can only be synthesized serially in theory, and the method of parallel synthesis according to the fragments destroys the continuity of the RNN hidden state between the fragments. However, the embodiment of the present disclosure creatively proposes a RNN hidden state prediction method, which provides an initial hidden state for each fragment through a hidden state prediction model, ensuring the approximate continuity of the hidden state between the fragments, so that while achieving parallel real-time synthesis, the synthesized sound quality is almost lossless. In addition, this RNN parallel real-time speech synthesis technology based on the hidden state continuity of the fragment can alleviate the error accumulation effect caused by RNN serial synthesis to a certain extent, and can effectively reduce the whistle phenomenon of synthesized speech.

[0035] refer to Figure 3 , which shows a schematic diagram of a process 300 of synthesizing multiple segments in parallel and in real time based on the hidden state continuity of the segments according to an embodiment of the present disclosure. The text 305 to be synthesized is divided into multiple segments 311, 312, 313, and then the hidden state prediction model 320 can predict that the initial hidden states of each segment 311, 312, 313 are h0 respectively. (1) 、h0 (2) 、h0 (3) It should be understood that although Figure 3 Only three segments are shown in FIG. 3 , however, the text 305 may be split into more segments.

[0036] Continue to refer Figure 3 , frame-level input features 341 of each segment can also be extracted from the text 305, for example, each frame can be 5 milliseconds, and then processed by the acoustic condition model 340 to generate sampling point level features 345. The acoustic condition model 340 can model the acoustic condition, and its input can be the linguistic features of the text, as shown below Figure 5An example structure of the acoustic condition model 340 is described.

[0037] The RNN-based speech synthesis model 330 realizes parallel synthesis of speech according to the initial hidden state and sampling point level features of each segment. Figure 3 As shown, at box 331, based on the initial hidden state of segment 311 and its sampling point level features, the synthesis for segment 311 is performed; at box 332, based on the initial hidden state of segment 312 and its sampling point level features, the synthesis for segment 312 is performed; at box 333, based on the initial hidden state of segment 313 and its sampling point level features, the synthesis for segment 313 is performed. In this way, parallel synthesis between multiple segments 311, 312, and 313 is achieved, which speeds up the speech synthesis speed without sacrificing the quality of the synthesized speech. After completing the synthesis of each segment, the speech of each segment can be smoothly connected to obtain the final complete speech.

[0038] It should be understood that the amount of computation introduced by the hidden state prediction model 320 of the embodiment of the present disclosure is very small compared to the amount of computation of the RNN, and can even be almost ignored. The RNN parallel speech synthesis method based on the continuous hidden state of the fragments of the embodiment of the present disclosure creatively solves the problem of RNN parallel reasoning, and while significantly improving the synthesis efficiency and meeting the real-time synthesis requirements, it ensures that the synthesis quality is almost lossless. In addition, compared with traditional parameter systems and splicing systems, the embodiment of the present disclosure is a high-quality speech synthesis system, which is suitable for the wide application of neural speech synthesis systems in the industry.

[0039] In some embodiments, for synthesis within a single segment, each segment may be synthesized serially in an autoregressive manner. For example, for the speech synthesis process in block 331, Figure 4 FIG. 4 is a schematic diagram showing a process 400 of serially synthesizing segments in an autoregressive manner according to an embodiment of the present disclosure.

[0040] Figure 4An example process of generating the output of the sampling points 410, 420, 430, etc. in the segment 311 is shown, where h0 is the initial hidden state of the segment 311, which is obtained from the hidden state prediction model 320 according to the embodiment of the present disclosure. In the process of generating the sampling points in each segment, the output of each sampling point needs to be generated based on the input features of the sampling point, the output of the previous sampling point, and the hidden state transmitted by the previous sampling point. For the first sampling point 410 in the segment 311, in addition to the sampling point feature i1, its input hidden state h0 can be the initial hidden state of the segment 311, its input previous sampling point input s0 can be 0, and its generated output is s1. Next, for the second sampling point 420, its input includes the hidden state h1 generated by the previous sampling point 410, the feature i2 of the sampling point, and the output s1 of the previous sampling point 410. Through the autoregressive serial synthesis in a single segment, the quality of speech synthesis of each segment can be guaranteed.

[0041] Figure 5 FIG. 5 is a schematic diagram showing an example architecture 500 of a speech synthesis system based on an RNN according to an embodiment of the present disclosure. Figure 5 As shown, the hidden state prediction model 320 may include 1 fully-connected (FC) layer and N bidirectional quasi-recurrent neural network (QRNN) layers, the acoustic conditional model 340 includes 2 bidirectional QRNN layers and 1 repeated upsampling layer, and the RNN-based speech synthesis model 330 may be implemented using 1 layer of gated recurrent unit (GRU). It should be understood that Figure 5 The architecture shown in is merely exemplary, and other suitable architectures may also be used in conjunction with the embodiments of the present disclosure.

[0042] refer to Figure 5 After obtaining the phoneme-level input features 511 and the frame-level input features 341 of each segment, the hidden state prediction model 320 predicts the initial hidden state 521 of each phoneme based on the phoneme-level input features 511, and then the initial hidden state of the first phoneme in the segment can be determined as the initial hidden state of the segment. Since the number of phonemes in a language is less than that of syllables, the hidden state prediction model 320 can be trained more easily using the phoneme-level input features, thereby predicting a more accurate initial hidden state.

[0043] The acoustic conditional model 340 obtains the sampling point level features 345 based on the frame level input features 341 by repeated upsampling. For example, if each frame feature corresponds to 80 speech sampling points, the frame level features are copied 80 times by repeated upsampling and used as the conditional input of the RNN-based speech synthesis model 330. The RNN-based speech synthesis model 330 synthesizes each segment based on the initial hidden state 521 and the sampling point level features 345, and then obtains the output synthesized speech 531.

[0044] The embodiments of the present disclosure add a hidden state prediction model based on the traditional RNN speech synthesis model. These two models can be trained together or separately. Figure 6 A schematic diagram of a separate training process 600 of a RNN-based speech synthesis system according to an embodiment of the present disclosure is shown. For example, the RNN speech synthesis model is first trained using training data, and after the RNN speech synthesis model is trained, the hidden state prediction model is trained using the training data and the trained RNN.

[0045] refer to Figure 6 , the training data 610 may include training text and corresponding training speech. Given the training data 610, frame-level input features 611, speech sampling points 612, and phoneme-level input features 613 may be extracted therefrom, wherein the frame-level input features 611 and the phoneme-level input features 613 may be obtained from the training text, and the speech sampling points 612 may be obtained from the training speech samples. In some embodiments, the frame-level input features 611 may include phoneme context, prosodic context, frame position, and fundamental frequency, etc., and the phoneme-level input features 613 may include text-level information such as phoneme context and prosodic context.

[0046] exist Figure 6 In the illustrated separate training process, the RNN-based speech synthesis model 330 is first trained at 620 using the frame-level input features 611 and the speech sampling points 612. Then, the phoneme-level hidden state 625 is obtained from the trained RNN-based speech synthesis model, for example, the initial hidden state of the first sampling point among the multiple sampling points corresponding to each phoneme can be determined as the phoneme-level hidden state of the phoneme.

[0047] In some embodiments, the phoneme level hidden state 625 and the phoneme level input feature 613 can be used to train the hidden state prediction model. Since the number of all phoneme samples in the training set may be relatively small, and the dimension of the hidden state is relatively high (e.g., 896 dimensions), if these high-dimensional hidden states are directly used as targets to train the hidden state prediction model, it is easy to cause the model to overfit. Therefore, in order to improve the training efficiency and model generalization ability, the high-dimensional phoneme level hidden state 625 can be clustered using a decision tree at box 630 to obtain phoneme level clustered hidden states 635, thereby reducing the number of hidden states. The clustered hidden state can be obtained by calculating the mean of all original hidden states in the class. Next, at box 640, the phoneme level input feature 613 and the corresponding phoneme level clustered hidden state 635 are used to train the hidden state prediction model.

[0048] In some embodiments, the hidden state prediction model predicts an initial hidden state for each phoneme, and then finds the corresponding phoneme boundary according to the selected segment to obtain the initial hidden state of each segment. In addition, the cross entropy loss function can be used to train the RNN-based speech synthesis model, while the L1 loss function can be used to train the hidden state prediction model.

[0049] Figure 7 FIG. 7 shows a block diagram of a parallel speech synthesis device 700 according to an embodiment of the present disclosure. Figure 7 As shown, the apparatus 700 includes a segment splitting module 710, a hidden state obtaining module 720, and a parallel speech synthesis module 730. The segment splitting module 710 is configured to split a text into multiple segments, the hidden state obtaining module 720 is configured to obtain multiple initial hidden states for a recurrent neural network of multiple segments based on a text, and the parallel speech synthesis module 730 is configured to synthesize multiple segments in parallel based on the multiple initial hidden states and the input features of the multiple segments.

[0050] In some embodiments, each of the multiple segments is any one of a phoneme, a syllable, and a prosodic word, and the parallel speech synthesis module 730 includes: a serial speech synthesis module, configured to serially synthesize each segment in an autoregressive manner based on the initial hidden state and input features of each segment.

[0051] In some embodiments, the hidden state acquisition module 720 includes: a phoneme level input feature determination module, configured to determine the phoneme level input features of each of a plurality of segments; and a hidden state prediction module, configured to predict the initial hidden state of each segment based on the phoneme level input features of each segment using a trained hidden state prediction model.

[0052] In some embodiments, the parallel speech synthesis module 730 includes: a frame-level input feature determination module, configured to determine the frame-level input features of each of the multiple segments; a sampling point-level feature acquisition module, configured to obtain sampling point-level features using an acoustic conditional model based on the frame-level input features; and a segment synthesis module, configured to synthesize each segment using a speech synthesis model based on a recurrent neural network based on the initial hidden state and sampling point-level features of each segment.

[0053] In some embodiments, the sampling point level feature acquisition module includes: a repeated upsampling module, configured to obtain sampling point level features through repeated upsampling as conditional input of the speech synthesis model.

[0054] In some embodiments, the device 700 also includes: a speech synthesis model training module, configured to use training data to train a speech synthesis model based on a recurrent neural network; and a hidden state prediction model training module, configured to use training data and the trained speech synthesis model to train a hidden state prediction model.

[0055] In some embodiments, the speech synthesis model training module includes: a first acquisition module, configured to obtain frame-level input features of training text and corresponding speech sampling points of training speech in training data, wherein the frame-level input features include at least one of phoneme context, prosodic context, frame position and fundamental frequency; and a first training module, configured to use the frame-level input features of the training text and the speech sampling points of the training speech to train the speech synthesis model.

[0056] In some embodiments, the hidden state prediction model training module includes: a second acquisition module, configured to obtain phoneme level input features of the training text, wherein the phoneme level input features include at least one of a phoneme context and a prosodic context; a third acquisition module, configured to obtain the phoneme level hidden state of each phoneme from a trained speech synthesis model; and a second training module, configured to train the hidden state prediction model using the phoneme level input features and the phoneme level hidden states.

[0057] In some embodiments, the second training module includes: a hidden state clustering module configured to cluster phoneme-level hidden states to generate phoneme-level clustered hidden states; and a third training module configured to train a hidden state prediction model using phoneme-level input features and phoneme-level clustered hidden states.

[0058] In some embodiments, the third obtaining module includes: a phoneme-level hidden state determining module configured to determine an initial hidden state of a first sampling point among a plurality of sampling points corresponding to each phoneme as a phoneme-level hidden state of each phoneme.

[0059] It should be understood that Figure 7 The segment splitting module 710, the hidden state acquisition module 720 and the parallel speech synthesis module 730 shown in the figure can be included in a single or multiple electronic devices. Moreover, it should be understood that Figure 7 The modules shown in the figure can perform steps or actions in the method or process of the embodiments of the present disclosure.

[0060] The fragment-based RNN parallel synthesis scheme of the embodiment of the present disclosure can overcome the problem of low efficiency of RNN serial synthesis, significantly improve the real-time rate of speech synthesis, and thus support real-time speech synthesis. In addition, in single-step recursive calculations, there is no need to specialize the model algorithm, so the acceleration cost is low. Compared with the traditional sub-scale batch sampling point generation strategy, the fragment-based RNN parallel synthesis technology of the embodiment of the present disclosure has the advantage of low latency. In scenarios where users have high requirements for synthesis response speed, the embodiments of the present disclosure have obvious advantages.

[0061] In addition, the embodiments of the present disclosure use a hidden state prediction model to provide an initial hidden state for each segment, which alleviates the problem of hidden state interruption between segments during parallel synthesis, ensures that the voice quality of parallel synthesis is basically equivalent to that of serial synthesis, and enables RNN to synthesize quickly without sacrificing synthesis performance. When training the hidden state prediction model, some embodiments of the present disclosure use a decision tree to cluster the hidden state of each phoneme, and use the clustered hidden state as the training target. In this way, the generalization ability of the hidden state prediction model can be improved.

[0062] In addition, compared with traditional parameter systems or splicing systems, the fragment-based RNN parallel synthesis system is a high-quality neural real-time speech synthesis system, which significantly exceeds traditional parameter systems or splicing systems in synthesis quality, and promotes the widespread application of neural speech synthesis systems in industry.

[0063] Figure 8 A schematic block diagram of an example device 800 that can be used to implement an embodiment of the present disclosure is shown. It should be understood that the device 800 can be used to implement the parallel speech synthesis device 700 described in the present disclosure. As shown in the figure, the device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or computer program instructions loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The CPU 801, ROM 802 and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0064] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0065] The processing unit 801 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions or steps of the method described above may be performed. Alternatively, in other embodiments, CPU 801 may be configured to perform the method in any other suitable manner (e.g., by means of firmware).

[0066] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0067] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0068] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0069] In addition, although each action or step is described in a specific order, this should be understood as requiring such action or step to be performed in the specific order shown or in a sequential order, or requiring all illustrated actions or steps to be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0070] Although the embodiments of the present disclosure have been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A parallel speech synthesis method, comprising: Split a text into multiple segments; Based on the piece of text, obtaining a plurality of initial hidden states for a recurrent neural network of the plurality of segments; as well as Based on the multiple initial hidden states and the input features of the multiple segments, the multiple segments are synthesized in parallel using a speech synthesis model based on a recurrent neural network.

2. The method of claim 1, wherein each of the plurality of segments is any one of a phoneme, a syllable, and a prosodic word, and wherein synthesizing the plurality of segments in parallel comprises: Based on the initial hidden state and input features of each segment, each segment is synthesized serially in an autoregressive manner.

3. The method according to claim 1, wherein obtaining a plurality of initial hidden states for the recurrent neural network of the plurality of segments comprises: determining a phoneme-level input feature for each segment of the plurality of segments; as well as Based on the phoneme-level input features of each segment, the trained latent state prediction model is used to predict the initial latent state of each segment.

4. The method according to claim 3, wherein synthesizing the plurality of segments in parallel using a speech synthesis model based on a recurrent neural network comprises: determining frame-level input features for each of the plurality of segments; Based on the frame-level input features, using an acoustic conditional model to obtain sampling point-level features; as well as Based on the initial hidden state of each segment and the sampling point level features, a speech synthesis model based on a recurrent neural network is used to synthesize each segment.

5. The method according to claim 4, wherein using the acoustic condition model to obtain sampling point level features comprises: The sampling point level features are obtained by repeated upsampling.

6. The method according to claim 1, further comprising: Use the training data to train a recurrent neural network-based speech synthesis model; as well as A hidden state prediction model is trained using the training data and the trained speech synthesis model.

7. The method according to claim 6, wherein training the speech synthesis model based on the recurrent neural network comprises: Obtaining frame-level input features of the training text in the training data and speech sampling points of the corresponding training speech, wherein the frame-level input features include at least one of a phoneme context, a prosodic context, a frame position, and a fundamental frequency; as well as The speech synthesis model is trained using the frame-level input features of the training text and the speech sampling points of the training speech.

8. The method of claim 7, wherein training the hidden state prediction model comprises: Obtaining a phoneme-level input feature of the training text, wherein the phoneme-level input feature includes at least one of a phoneme context and a prosodic context; Obtaining a phoneme-level hidden state for each phoneme from the trained speech synthesis model; as well as The latent state prediction model is trained using the phoneme-level input features and the phoneme-level latent states.

9. The method according to claim 8, wherein training the hidden state prediction model further comprises: clustering the phoneme-level hidden states to generate phoneme-level clustered hidden states; as well as The latent state prediction model is trained using the phoneme-level input features and the phoneme-level clustered latent states.

10. The method of claim 8, wherein obtaining a phoneme-level hidden state of each phoneme from the trained speech synthesis model comprises: An initial hidden state of a first sampling point among a plurality of sampling points corresponding to each phoneme is determined as a phoneme-level hidden state of each phoneme.

11. A parallel speech synthesis device, comprising: A segment splitting module, configured to split a text into multiple segments; A hidden state acquisition module is configured to obtain a plurality of initial hidden states for a recurrent neural network of the plurality of segments based on the plurality of segments of text; as well as The parallel speech synthesis module is configured to synthesize the multiple segments in parallel using a speech synthesis model based on a recurrent neural network based on the multiple initial hidden states and the input features of the multiple segments.

12. The apparatus according to claim 11, wherein each of the plurality of segments is any one of a phoneme, a syllable, and a prosodic word, and the parallel speech synthesis module comprises: The serial speech synthesis module is configured to serially synthesize each segment in an autoregressive manner based on the initial hidden state and input features of each segment.

13. The apparatus according to claim 11, wherein the hidden state acquisition module comprises: a phoneme-level input feature determination module configured to determine a phoneme-level input feature of each of the plurality of segments; as well as The hidden state prediction module is configured to predict an initial hidden state of each segment using a trained hidden state prediction model based on the phoneme-level input features of each segment.

14. The apparatus according to claim 13, wherein the parallel speech synthesis module comprises: a frame-level input feature determination module configured to determine a frame-level input feature for each of the plurality of segments; A sampling point level feature acquisition module is configured to obtain sampling point level features using an acoustic conditional model based on the frame level input features; as well as The segment synthesis module is configured to synthesize each segment using a speech synthesis model based on a recurrent neural network based on an initial hidden state of each segment and the sampling point level features.

15. The device according to claim 14, wherein the sampling point level feature acquisition module comprises: The repeated upsampling module is configured to obtain the sampling point level features through repeated upsampling.

16. The apparatus according to claim 11, further comprising: A speech synthesis model training module is configured to train a speech synthesis model based on a recurrent neural network using the training data; as well as The hidden state prediction model training module is configured to train a hidden state prediction model using the training data and the trained speech synthesis model.

17. The apparatus according to claim 16, wherein the speech synthesis model training module comprises: A first obtaining module is configured to obtain frame-level input features of the training text in the training data and speech sampling points of the corresponding training speech, wherein the frame-level input features include at least one of phoneme context, prosodic context, frame position and fundamental frequency; as well as The first training module is configured to train the speech synthesis model using the frame-level input features of the training text and the speech sampling points of the training speech.

18. The apparatus according to claim 17, wherein the hidden state prediction model training module comprises: A second obtaining module is configured to obtain a phoneme-level input feature of the training text, wherein the phoneme-level input feature includes at least one of a phoneme context and a prosodic context; A third acquisition module is configured to obtain a phoneme-level hidden state of each phoneme from the trained speech synthesis model; as well as A second training module is configured to train the latent state prediction model using the phoneme-level input features and the phoneme-level latent states.

19. The apparatus according to claim 18, wherein the second training module comprises: a hidden state clustering module configured to cluster the phoneme-level hidden states to generate phoneme-level clustered hidden states; as well as A third training module is configured to train the hidden state prediction model using the phoneme-level input features and the phoneme-level clustered hidden states.

20. The apparatus according to claim 18, wherein the third obtaining module comprises: The phoneme-level hidden state determination module is configured to determine an initial hidden state of a first sampling point among a plurality of sampling points corresponding to each phoneme as a phoneme-level hidden state of each phoneme.

21. An electronic device, comprising: one or more processors; as well as A memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, enable the electronic device to implement the method according to any one of claims 1-10.

22. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images