Neural network-based synthesized speech method, system, device, and storage medium

Abstract

This invention relates to the field of artificial intelligence technology, and proposes a speech synthesis method, system, device, and storage medium based on neural networks. The method includes: acquiring target text information; inputting the target text information into a speech synthesis model for processing to obtain the target Mel spectrum corresponding to the target text information. The speech synthesis model is an autoregressive speech synthesis model, trained using training text and speech tags. During training, the speech synthesis model aligns the target text information and the target speech based on the linear relationship between the training text and the speech tags; and generates the target speech information based on the target Mel spectrum. This invention improves the training efficiency of the autoregressive speech synthesis model, mitigates word omission and repetition issues, and enhances the robustness of the speech synthesis method by introducing prior information about the linear relationship between the training text and the speech tags during the speech synthesis model training process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a neural network-based synthesized speech method, system, device and storage medium. BACKGROUND

[0002] Speech synthesis, also known as text-to-speech (TTS), is a technology that can convert any input text into corresponding speech. With the development of deep learning, current end-to-end speech synthesis technology can already produce high-quality speech. Compared with the two traditional speech synthesis methods of parametric speech synthesis and spliced speech synthesis, the end-to-end speech synthesis method does not need to model the speech complexly, and can produce more natural speech.

[0003] However, existing end-to-end speech synthesis models are usually divided into autoregressive speech synthesis models and non-autoregressive speech synthesis models. Among them, the autoregressive speech synthesis model refers to the output of the model being output step by step, i.e. the output of each step depends on the previous output. Therefore, whether in the training process or in the actual use process, this model is very time-consuming and has low efficiency in synthesizing speech. The non-autoregressive model refers to the output of the model belonging to the full parallel output. Although this model synthesizes speech quickly, the quality of the speech synthesized by the model is very low due to the need for distillation.

[0004] For common autoregressive speech synthesis models, the model can generate high-natural speech within a text set. These autoregressive speech models will encounter pronunciation problems, have out-of-text set single-word omission and repetition problems, and also have high training costs, requiring a large amount of data and computing power. SUMMARY

[0005] The present application provides a neural network-based speech synthesis method, system, device and storage medium, which mainly aims to promote fast convergence in the speech model training process, improve single-word omission and repetition problems, and effectively improve the robustness of the speech synthesis method.

[0006] In a first aspect, an embodiment of the present application provides a neural network-based speech synthesis method, comprising:

[0007] obtaining target text information;

[0008] inputting the target text information into a speech synthesis model for processing to obtain a target mel spectrum corresponding to the target text information, wherein the speech synthesis model is an autoregressive speech synthesis model, the speech synthesis model is obtained by training a training text and a speech label, and the speech synthesis model realizes alignment between the target text information and a target speech according to a linear relationship between the training text and the speech label in a training process;

[0009] generating the target speech based on the target mel spectrum.

[0010] Preferably, the speech synthesis model includes an encoder and an attention module, and the speech synthesis model realizes alignment between the target text information and the target speech according to a linear relationship between the training text and the speech label in a training process, including:

[0011] inputting the training text into the encoder to output an encoding sequence corresponding to the training text;

[0012] projecting the encoding sequence corresponding to the training text and a preset attention feature into the attention module to obtain an attention weight matrix corresponding to the training text;

[0013] obtaining a training error according to the attention weight matrix corresponding to the training text and a preset loss function, adjusting network parameters of the attention module if the training error is outside a preset error threshold, and retraining the attention module according to the adjusted parameters until the training error obtained by retraining is within the preset error threshold, and a difference between an element on a diagonal line position in the attention weight matrix corresponding to the training sample obtained by recalculation and an element outside the diagonal line position is greater than a preset difference threshold, and finally obtaining a trained encoder and a trained attention module, wherein the preset loss function is obtained according to a probability mass function of the training text and the speech label and the attention weight matrix.

[0014] Preferably, the speech synthesis model further includes a decoder, and the inputting the target text information into the speech synthesis model for processing to obtain the target mel spectrum corresponding to the target text information includes:

[0015] inputting the target text information into the trained encoder to output an encoding sequence corresponding to the target text information;

[0016] projecting the encoding sequence corresponding to the target text information and a preset attention feature into the trained attention module to obtain an attention weight matrix corresponding to the target text information, and obtaining a context vector corresponding to the target text information according to the attention weight matrix corresponding to the target text information and the encoding sequence corresponding to the target text information.

[0017] inputting the attention weight matrix corresponding to the target text information and the context vector corresponding to the target text information into the decoder to obtain a target mel spectrum corresponding to the target text information.

[0018] Preferably, the calculation formula of the preset loss function is as follows:

[0019]

[0020] W=1-f B (j,ωj,ω(T-i+1)),

[0021] ω=0.2,

[0022] wherein L represents the preset loss function, a represents the attention weight matrix corresponding to the training text, i represents the character position in the training text, j represents the position of a frame in the predicted speech, T represents the total number of frames of the predicted speech, f B represents the probability mass function of the beta-binary distribution, and B represents the beta function.

[0023] Preferably, the preset attention feature is a position attention feature or a content attention feature.

[0024] Preferably, the speech synthesis model is one of Tacotron, a speech synthesis model optimized based on Tacotron, DeepVoice, and a speech synthesis model optimized based on DeepVoice.

[0025] Preferably, the obtaining of the target text information comprises:

[0026] character embedding is performed on the input text data, a fixed-dimension vector is used to represent the input text data, then the input text data is sequentially passed through a Pre-Net and a CBHG subnetwork to obtain the target text information.

[0027] Preferably, the generating of the target speech information based on the target mel spectrum comprises:

[0028] The target mel spectrum is converted into a linear spectrogram through a post-processing network, and the linear spectrogram is restored into a speech waveform through a Griffin-Lim algorithm.

[0029] In a second aspect, an embodiment of the present application provides a speech synthesis system based on a neural network, comprising:

[0030] an obtaining module configured to obtain target text information;

[0031] The processing module is configured to input the target text information into a speech synthesis model for processing to obtain a target mel spectrum corresponding to the target text information, wherein the speech synthesis model is a self-recurrent speech synthesis model, the speech synthesis model is trained by a training text and a speech label, and the speech synthesis model realizes alignment between the target text information and target speech according to a linear relationship between the training text and the speech label in a training process.

[0032] The generating module is configured to generate the target speech based on the target mel spectrum.

[0033] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the speech synthesis method based on a neural network when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program implements the steps of the speech synthesis method based on a neural network when executed by a processor.

[0035] The speech synthesis method, system, device and storage medium based on a neural network provided by the present application introduce prior information that a linear relationship exists between a training text and a speech label in a training process of a speech synthesis model, supervise parameters of the speech synthesis model through the prior information, effectively reduce the number of iterations of the speech synthesis model, accelerate the convergence speed of the speech synthesis model in the training process, and improve the training efficiency of the self-recurrent speech synthesis model. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 An application scenario diagram of the speech synthesis method based on a neural network provided by an embodiment of the present application;

[0037] Figure 2 A flowchart of the speech synthesis method based on a neural network provided by an embodiment of the present application;

[0038] Figure 3 A structure diagram of the speech synthesis system based on a neural network provided by an embodiment of the present application;

[0039] Figure 4A structural schematic diagram of a computer device provided by an embodiment of the present application is shown in the figure.

[0040] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0041] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0042] Figure 1 An application scenario diagram of the speech synthesis method based on the neural network provided by an embodiment of the present application is shown in the figure. Figure 1 As shown in the figure, a user inputs target text information on a client, the client sends the target text information to a server after receiving the target text information, and the server executes the speech synthesis method based on the neural network after receiving the target text information to obtain target speech information.

[0043] It should be noted that the server can be implemented by an independent server or a server cluster composed of multiple servers. The client can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The client and the server can be connected through Bluetooth, a universal serial bus (USB) or other communication connection modes, and the embodiments of the present application do not limit the same.

[0044] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer to perceive an environment, acquire knowledge and use the knowledge to obtain optimal results.

[0045] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology and several major directions such as machine learning and deep learning.

[0046] Figure 2 A flowchart of the speech synthesis method based on the neural network provided by an embodiment of the present application is shown in the figure. Figure 2 The method comprises the following steps.

[0047] S210, acquiring target text information;

[0048] Firstly, target text information is acquired, the target text information is text information needing to be synthesized into speech, the target text information contains text data, the text data can be a word, a phrase, a group of words, a sentence and a combination of multiple sentences, the above are only exemplary descriptions, and the format and content of the target text information are not limited in the embodiment of the application, and details can be determined according to actual conditions.

[0049] In the process of specific implementation, the terminal processing text information acquires target text information when detecting a speech synthesis instruction. The speech synthesis instruction is an instruction for instructing the terminal to synthesize speech according to the acquired target text information. The speech synthesis instruction can be triggered by a user, such as the user clicking a speech synthesis option in the terminal. The acquired target text information can be text information uploaded to the terminal by the user, or the terminal can acquire a text file corresponding to a file identifier according to the file identifier contained in the speech synthesis instruction, and extract text information in the text file.

[0050] S220, inputting the target text information into a speech synthesis model for processing to obtain a target mel spectrum corresponding to the target text information, wherein the speech synthesis model is an autoregressive speech synthesis model, the speech synthesis model is obtained by training a training text and a speech label, and the speech synthesis model realizes alignment between the target text information and a target speech according to a linear relationship between the training text and the speech label in a training process;

[0051] Then the target text information is input into the speech synthesis model, and the target mel spectrum corresponding to the target text information can be obtained. The mel spectrum is a general frequency spectrum plus a mel filter function. This step is to simulate the sensitivity of human ear hearing to actual frequency.

[0052] The autoregressive speech synthesis model is a neural network model, and the basic unit of the neural network model is a neuron. Through training and learning of the neuron, the neural network model can be finally applied to practice.

[0053] The speech synthesis model commonly has an autoregressive speech synthesis model and a non-autoregressive speech synthesis model. The speech synthesis model in the embodiment of the application specifically refers to the autoregressive speech synthesis model. The autoregressive speech synthesis model refers to that the output of the model is output step by step, that is, the output of each step depends on the previous output. The autoregressive speech synthesis model usually includes an encoder for mapping input text to hidden states, a decoder for converting hidden states into speech features, and an alignment mechanism between the encoder and the decoder, and the alignment mechanism depends on an attention module.

[0054] The traditional autoregressive speech synthesis model can generate high-natural speech within a text set, but the autoregressive speech model has pronunciation problems, has out-of-text set single word omission and repetition problems, and has a high training cost, requiring a large amount of data and computing power. In view of the problems in the traditional method, the embodiment of the application firstly proposes to integrate the prior information that the training text and the speech label are in a linear relationship into the training process of the speech synthesis model. Since the character information in the training text is extracted one by one by the autoregressive speech synthesis model in the order of characters, and the output speech is also synthesized one by one according to the character information, the position of the character information and the number of speech frames should be a one-to-one correspondence. By using this principle, the character information and the number of speech frames are aligned in the training process of the autoregressive speech synthesis model. By supervising the parameters of the speech synthesis model through this prior information, the iteration number of the speech synthesis model can be effectively reduced, and the convergence speed of the speech synthesis model in the training process can be accelerated, and the training efficiency of the autoregressive speech synthesis model can be improved. And through the prior information that the training text and the speech label are in a linear relationship, by ensuring that the text position and the number of speech frames are monotonically increasing in the training process of the speech synthesis model, the alignment accuracy between the training text and the output speech can be improved, thereby improving the single word omission and repetition problems, and improving the robustness of the speech synthesis method.

[0055] It should be noted that the autoregressive speech synthesis model in the embodiment of the application belongs to a kind of neural network, and before using the autoregressive speech synthesis model, it also needs to be trained or updated training, and the autoregressive speech synthesis model is trained by obtaining sample and label. The training process of the autoregressive speech synthesis model can be divided into three steps: defining the structure of the autoregressive speech synthesis model and the output result of forward propagation;According to the process described above, define the loss function and the algorithm of back propagation optimization;Finally, generate a conversation and repeatedly run the back propagation optimization algorithm on the training data.

[0056] Wherein, neuron is the smallest unit of neural network, a neuron can have multiple inputs and an output, the input of each neuron can be the output of other neurons, or the input of the whole neural network. The output of the neural network is the weighted sum of all neuron inputs, and the weight of different inputs is the neuron parameter, and the optimization process of the neural network is the process of optimizing the value of the neuron parameter.

[0057] The effect of the neural network and the optimization target are defined by a loss function, which gives a formula for the gap between the output of the neural network and the true label. Supervised learning is a way of training a neural network, and the idea is that the result given by the neural network should be as close as possible to the true answer (i.e. the label) on the labeled data set with known answers. By adjusting the parameters in the neural network to fit the training data, the neural network can provide prediction ability for unknown samples.

[0058] The back propagation algorithm implements an iterative process. At the beginning of each iteration, a part of the training data is taken, and the prediction result of the neural network is obtained by the forward propagation algorithm. Because the training data has correct answers, the gap between the prediction result and the correct answer can be calculated. Based on this gap, the back propagation algorithm updates the value of the neural network parameters accordingly, so that it is closer to the true answer.

[0059] After the training process is completed by the above method, the trained autoregressive speech synthesis model can be used for application.

[0060] S230, generating the target speech based on the target mel spectrum.

[0061] The target mel spectrum is converted into the target speech information corresponding to the target text information. For example, the target mel spectrum can be input into a trained neural vocoder for processing to obtain the speech information corresponding to the target text information, wherein the speech information includes the audio corresponding to the target text information.

[0062] For example, the trained vocoder can be a trained WaveGlow model. The WaveGlow model is a generative model that generates audio from distribution sampling. In the training process, the WaveGlow model models the distribution of audio samples conditioned on the mel spectrum graph, i.e. multiple sample mel spectrum graphs and the speech information corresponding to each sample mel spectrum graph. In the training process, the network parameters of the WaveGlow model are adjusted until a trained WaveGlow model is obtained.

[0063] The WaveGlow model is used to convert the target mel spectrum into the final audio, which improves the quality of the final synthesized speech. Moreover, the WaveGlow model is a fully parallel network, which enables the present scheme to realize fully parallel end-to-end speech synthesis.

[0064] In the above embodiment, the terminal obtains target text information, inputs the target text information into a trained speech synthesis model for processing to obtain a target mel spectrum corresponding to the target text information, the spectrum generation model is an autoregressive speech synthesis model, and the speech information corresponding to the target text information is generated based on the target mel spectrum. In the above scheme, the prior information that the linear relationship between the training text and the speech label is introduced in the process of training the speech synthesis model, the parameters of the speech synthesis model are supervised through the prior information, which can effectively reduce the iteration number of the speech synthesis model and speed up the convergence speed of the speech synthesis model in the training process, and improve the training efficiency of the autoregressive speech synthesis model. And through the prior information that the linear relationship between the training text and the speech label, by ensuring that the text position and the number of speech frames are monotonically increasing in the training process of the speech synthesis model, the alignment accuracy between the training text and the output speech can be improved, thereby improving the single word omission and repetition problem, and improving the robustness of the speech synthesis method.

[0065] On the basis of the above embodiment, preferably, the speech synthesis model comprises an encoder and an attention module, and the alignment between the target text information and the target speech is realized according to the linear relationship between the training text and the speech label in the training process of the speech synthesis model, comprising:

[0066] The training text is input into the encoder to output an encoding sequence corresponding to the training text;

[0067] The encoding sequence corresponding to the training text and a preset attention feature are projected into the attention module to obtain an attention weight matrix corresponding to the training text;

[0068] According to the attention weight matrix corresponding to the training text and a preset loss function, a training error is obtained, if the training error is outside the preset error threshold, the network parameters of the attention module are adjusted, and the attention module is retrained according to the adjusted parameters, until the training error obtained by retraining is within the preset error threshold, and the difference between the elements on the diagonal position and the elements outside the diagonal position in the attention weight matrix corresponding to the training sample obtained by re-computing is greater than a preset difference threshold, finally obtaining the trained encoder and the trained attention module, wherein the preset loss function is obtained according to the probability mass function of the training text and the speech label, and the attention weight matrix.

[0069] Specifically, the speech synthesis model in the embodiment of the present application comprises three parts of an encoder, an attention module and a decoder, but only the encoder and the attention module need to be trained when training the speech synthesis model, and the speech synthesis model comprises the trained encoder and the trained attention module after training, and the three parts of the encoder, the attention module and the decoder are all used when the speech synthesis model is applied.

[0070] In the training process, the training text is encoded by the encoder to obtain the encoding sequence corresponding to the training text, and the encoder can comprise three layers of convolutional neural networks (CNN) and one layer of long short-term memory networks (LSTM). Each layer of CNN is connected with one layer of rectified linear units (ReLu), one layer of batch normalization and one layer of dropout. The function of the LSTM network is to capture the relationship between the contexts in the text information.

[0071] In the embodiment of the present application, the encoder can be an encoder in a Tacotron framework, which is a prior art. It can also be composed of a Pre-net preprocessing network and a CBHG module. The Pre-net is used for preprocessing the input text, and the CBHG module is composed of a one-dimensional convolution filter group, a residual connection, a multi-layer highway network and a bidirectional gated recurrent unit (GRU) network in sequence. The one-dimensional convolution filter group is a convolution layer composed of m one-dimensional filters with different sizes, and the sizes of the filters are 1, 2, 3, …, m. The use of residual connection can alleviate the gradient dispersion problem caused by too deep neural network layers, so that it can ensure that too much information input previously will not be lost after multi-layer convolution. The highway network is used to alleviate the overfitting problem caused by deepening the network and reduce the training difficulty of deep network. Finally, the GRU is used to obtain the bidirectional feature sequence.

[0072] The training text is encoded by the encoder to obtain the encoding sequence corresponding to the training text, and the specific calculation formula is as follows:

[0073]

[0074] Wherein, x represents the input text, j represents the character of the input text, L represents the length of the input text, and h represents the encoding sequence.

[0075] The encoding sequence corresponding to the training text and the preset attention feature are projected into an attention module to obtain an attention weight matrix corresponding to the training text.

[0076] In the embodiment of the application, the content and position sensitive attention mechanism is taken as an example for illustration, which can consider the content and position of the input phoneme simultaneously, and can make the cumulative attention weight after the previous decoding process be regarded as an additional feature, so that the model keeps consistency when advancing along the input sequence, and reduces the problems of subsequence omission or repetition that may occur during decoding.

[0077] In the embodiment of the application, the position feature of the position sensitive attention mechanism is obtained by convolution of 32 one-dimensional convolution kernels with a length of 31, and the encoding sequence output in the previous step and the position feature are projected into a 128-dimensional hidden layer representation to obtain an attention weight, and then a context vector is obtained according to the attention weight matrix and the encoding sequence.

[0078] The encoding sequence corresponding to the training text and the preset attention feature are projected into an attention module to obtain an attention weight matrix corresponding to the training text.

[0079] s i =RNN att (s i ,c i-1 ,y i-1 ),

[0080] a i =Attention(s i ,…),c i =∑a i,j h j 。

[0081] Wherein, a represents the attention weight matrix, s is a vector, and y represents the output speech.

[0082] The attention weight matrix corresponding to the training text and the context vector corresponding to the training text are then input into a decoder to obtain a context vector corresponding to the training text, and the decoder is an autoregressive recurrent neural network that predicts the output spectrogram from the encoding sequence output by the encoder, one frame at a time. The frequency spectrum frame predicted in the previous step is first transmitted into a double-layer fully connected preprocessing network pre-net composed of 256 hidden ReLU units per layer.

[0083] The output of the pre-net and the attention context vector are spliced together and transmitted to a two-layer stacked unidirectional neural network composed of 1024 units, the output of the neural network is again spliced with the attention context vector, and then a linear transformation projection is performed to predict the target spectrum frame.

[0084] The predicted target spectrum frame is transmitted through a 5-layer convolutional network to predict a residual superimposed on the spectrum frame before convolution, so as to improve the entire process of spectrum reconstruction. Each layer of the network is composed of 512 5x1 convolution kernels and a batch normalization process, and the batch normalization process of each layer is followed by a tanh activation function, except for the last layer of convolution.

[0085] In the specific implementation process, the output of the decoder is spliced with the attention context vector, projected into a scalar, and then transmitted to a sigmoid activation function to predict the probability of whether the output sequence has been completed.

[0086] When the probability value is greater than or equal to a preset end threshold, it indicates that the prediction is completed, and the next step is performed.

[0087] The convolutional layer in the network is regularized by dropout with a probability of 0.5, and the LSTM layer is regularized by zoneout with a probability of 0.1. In order to bring some changes to the output result during inference, dropout with a probability of 0.5 is only applied to the pre-net of the autoregressive decoder.

[0088] For a training, after the attention weight matrix of the training is calculated, the training error is calculated according to the attention weight matrix and the preset loss function. In the embodiment of the present application, considering that there is a linear relationship between the text position j and the speech frame number i, i.e. j≈k×i, under normal training conditions, the attention weight matrix a i*j Should be close to a diagonal matrix, here the so-called close to is that the difference between the elements on the diagonal line and the elements on other positions in the attention weight matrix a i*j The elements on the diagonal line are larger and the elements on other positions are smaller, and the elements on the diagonal line are much larger than the elements on other positions. In the embodiment of the present application, it is defined that the attention weight matrix is close to a diagonal matrix by the difference between the elements on the diagonal line and the elements on other positions in the attention weight matrix being greater than a preset difference threshold, and the preset difference threshold is determined according to actual conditions in the embodiment of the present application, which is not limited here.

[0089] On the basis of the above embodiment, preferably, the speech synthesis model further comprises a decoder, and the processing of inputting the target text information into the speech synthesis model to obtain the target mel spectrum corresponding to the target text information comprises:

[0090] input the target text information into the trained encoder, and output an encoding sequence corresponding to the target text information;

[0091] project the encoding sequence corresponding to the target text information and a preset attention feature into the trained attention module, obtain an attention weight matrix corresponding to the target text information, and obtain a context vector corresponding to the target text information according to the attention weight matrix corresponding to the target text information and the encoding sequence corresponding to the target text information;

[0092] input the attention weight matrix corresponding to the target text information and the context vector corresponding to the target text information into the decoder, and obtain a target mel spectrum corresponding to the target text information.

[0093] In the application process of the speech synthesis model, in addition to the trained decoder and the trained attention module, the encoder is also needed. Here, the target text information is input into the speech synthesis model for processing to obtain a target mel spectrum corresponding to the target text information. The specific process is as follows:

[0094] input the target text information into the trained encoder to obtain an encoding sequence corresponding to the target text information, then project the encoding sequence corresponding to the target text information and a preset attention feature into the trained attention module to obtain an attention weight matrix corresponding to the target text information, and obtain a context vector corresponding to the target text information according to the attention weight matrix corresponding to the target text information and the encoding sequence corresponding to the target text information. The specific implementation process is the same as the above training process, and details are referred to the above training process.

[0095] Then input the attention weight matrix corresponding to the target text information and the context vector corresponding to the target text information into the decoder to obtain a target mel spectrum corresponding to the target text information, and the specific calculation formula is as follows:

[0096] d i =RNN dec (d i-1 ,c i ,s i )。

[0097] Then, according to the target mel spectrum, a target speech is obtained, and the calculation formula is as follows:

[0098] y i =f(d i )。

[0099] Wherein, d represents the target mel spectrum, and y represents the target speech.

[0100] On the basis of the above-mentioned embodiments, the preset loss function in the embodiments of the present application is obtained according to a probability mass function of a training text and a phonetic label and an attention weight matrix, and the specific calculation formula is as follows:

[0101]

[0102] W = 1 - f B (j, ωj, ω(T-i+1)),

[0103] ω = 0.2,

[0104] Wherein, L represents the preset loss function, i represents the character position in the training text, j represents the position of the frame in the predicted phonetic, T represents the total frame number of the predicted phonetic, f B represents the probability mass function of the beta-binary distribution, the value range is 0 to 1, when the attention weight matrix is close to the diagonal line value, it tends to 1, and far away from the diagonal line, it tends to 0, like a diagonal matrix.

[0105] Specifically,

[0106] Wherein, B represents the beta function.

[0107] It should be noted that in the process of training the speech synthesis model, a represents the attention weight matrix corresponding to the training text.

[0108] After calculating the training error according to the above formula, if the training error is outside the preset error threshold, it means that the training result does not meet the requirements, and further training is needed, then the network parameters of the attention module are adjusted, and the attention module after adjusting the network parameters is retrained according to the same method, until the training error obtained by retraining is within the preset error threshold, and the difference between the elements on the diagonal line position and the elements on other positions in the attention weight matrix is greater than the preset difference threshold, that is, the newly calculated attention weight matrix is close to the diagonal matrix, and the training is completed.

[0109] On the basis of the above-mentioned embodiments, preferably, the preset attention feature is a position attention feature or a content attention feature.

[0110] The preset attention feature in the embodiments of the present application can be a position attention feature or a content attention feature, and the position attention feature calculation formula is as follows:

[0111]

[0112] Wherein, s i is the hidden state of the decoder recurrent neural network at the i-th moment, h j is the j-th output of the encoder, and fi,j represents the convolution output of the accumulated attention weight at the i moment, b represents a bias term, and the initial value is a 0 vector, v a , W, V and U represent weight matrices of different network layers, represents the transpose of v a .

[0113] On the basis of the above embodiment, preferably, the speech synthesis model is one of Tacotron, a speech synthesis model optimized based on Tacotron, DeepVoice, and a speech synthesis model optimized based on DeepVoice.

[0114] The speech synthesis model in the embodiment of the application includes Tacotron, a speech synthesis model optimized based on Tacotron, DeepVoice, and a speech synthesis model optimized based on DeepVoice, and the speech synthesis model optimized based on Tacotron includes Tacotron2 and the like, and the speech synthesis model optimized based on DeepVoice includes DeepVoice3.

[0115] On the basis of the above embodiment, preferably, the obtaining of the target text information includes:

[0116] Character embedding is performed on the input text data, a fixed-dimension vector is used to represent the input text data, and then the input text data is sequentially passed through a Pre-Net and a CBHG subnetwork to obtain the target text information.

[0117] Specifically, the text feature extraction module first performs character embedding on the input text data, that is, uses a fixed-dimension vector to represent the text characters, and then sequentially passes the input text data through a Pre-Net and a CBHG subnetwork to obtain text feature data.

[0118] On the basis of the above embodiment, preferably, the generating of the target speech information based on the target mel spectrum includes:

[0119] The target mel spectrum is converted into a linear spectrogram through a post-processing network, and the linear spectrogram is restored into a speech waveform output through a Griffin-Lim algorithm.

[0120] Specifically, the output of the decoder is converted into a linear spectrogram through a post-processing network, and the linear spectrogram is restored into a speech waveform output through a Griffin-Lim algorithm.

[0121] In summary, the embodiment of the present application provides a speech synthesis system based on a neural network, a new loss function is proposed by introducing prior information between input text and speech. The loss function supervises the attention weight in the training process of the autoregressive speech synthesis model by the feature that the probability mass function of the beta-binary distribution is like a diagonal matrix, and forces the attention weight of the autoregressive speech synthesis model to be close to a diagonal matrix.

[0122] The experimental results show that it effectively reduces the number of model iterations, accelerates the convergence speed of the alignment mechanism, and improves the training efficiency of the autoregressive speech synthesis model. At the same time, aiming at the defect that the traditional alignment mechanism is poor in generalization ability for long text and text set, it significantly improves the problems of single word omission and repetition, and improves the robustness of the autoregressive speech synthesis method.

[0123] Figure 3 A structure diagram of a speech synthesis system based on a neural network provided by the embodiment of the present application is shown in Figure 3 As shown in the figure, the system includes a processing module 310, a processing module 320 and a generation module 330, wherein:

[0124] The acquisition module 310 is used to acquire target text information;

[0125] The processing module 320 is used to input the target text information into a speech synthesis model for processing to obtain a target mel spectrum corresponding to the target text information, wherein the speech synthesis model is an autoregressive speech synthesis model, the speech synthesis model is obtained by training text and speech labels, and the speech synthesis model realizes the alignment between the target text information and the target speech according to the linear relationship between the training text and the speech label in the training process;

[0126] The generation module 330 is used to generate the target speech based on the target mel spectrum.

[0127] The embodiment of the present application is a system corresponding to the above method, and the specific implementation process is the same as the above method embodiment. For details, please refer to the above method embodiment. The system embodiment will not be repeated here.

[0128] On the basis of the above embodiment, preferably, the speech synthesis model includes an encoder and an attention module, and the processing module includes a first unit, a second unit and a third unit, wherein:

[0129] The first unit is used to input the training text into the encoder to output an encoding sequence corresponding to the training text;

[0130] The second unit is configured to project the encoding sequence corresponding to the training text and the preset attention feature into the attention module to obtain an attention weight matrix corresponding to the training text.

[0131] The third unit is configured to obtain a training error according to the attention weight matrix corresponding to the training text and a preset loss function, adjust network parameters of the attention module if the training error is outside a preset error threshold, and retrain the attention module according to the adjusted parameters until the training error obtained by retraining is within the preset error threshold, and a difference between an element at a diagonal position in the attention weight matrix corresponding to the training sample and an element outside the diagonal position is greater than a preset difference threshold, and finally obtain a trained encoder and a trained attention module, wherein the preset loss function is obtained according to a probability mass function of the training text and the speech label and the attention weight matrix.

[0132] On the basis of the above-mentioned embodiments, preferably, the speech synthesis model further comprises a decoder, and the generation module comprises a fourth unit, a fifth unit and a sixth unit, wherein:

[0133] The fourth unit is configured to input the target text information into the trained encoder to output an encoding sequence corresponding to the target text information.

[0134] The fifth unit is configured to project the encoding sequence corresponding to the target text information and a preset attention feature into the trained attention module to obtain an attention weight matrix corresponding to the target text information, and obtain a context vector corresponding to the target text information according to the attention weight matrix corresponding to the target text information and the encoding sequence corresponding to the target text information.

[0135] The sixth unit is configured to input the attention weight matrix corresponding to the target text information and the context vector corresponding to the target text information into the decoder to obtain a target mel spectrum corresponding to the target text information.

[0136] On the basis of the above-mentioned embodiments, preferably, a calculation formula of the preset loss function is as follows:

[0137]

[0138] W=1-f B (j,ωj,ω(T-i+1)),

[0139] ω=0.2,

[0140] Wherein, L represents the preset loss function, a represents the attention weight matrix corresponding to the training text, i represents the character position in the training text, j represents the position of the frame in the predicted voice, T represents the total number of frames of the predicted voice, f B denotes the probability mass function of the Beta-Binary distribution, B denotes the Beta function.

[0141] On the basis of the above embodiment, preferably, the preset attention feature is a position attention feature or a content attention feature.

[0142] On the basis of the above embodiment, preferably, the speech synthesis model is one of Tacotron, a speech synthesis model optimized based on Tacotron, DeepVoice, and a speech synthesis model optimized based on DeepVoice.

[0143] On the basis of the above embodiment, preferably, the obtaining module comprises an embedding unit, wherein:

[0144] The embedding unit is configured to perform character embedding on the input text data, use a fixed-dimension-size vector to represent the input text data, and then sequentially pass through a Pre-Net and a CBHG subnetwork to obtain the target text information.

[0145] On the basis of the above embodiment, preferably, the generating module comprises an output unit, wherein:

[0146] The output unit is configured to convert the target mel-frequency spectrum into a linear spectrogram through a post-processing network, and restore the linear spectrogram into a voice waveform through a Griffin-Lim algorithm to output.

[0147] The above various modules in the speech synthesis system based on the neural network can be realized by software, hardware, and combinations thereof, in whole or in part. The above various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above various modules.

[0148] Figure 4 A structural schematic diagram of a computer device is provided for the embodiments of the present application. The computer device can be a server, and its internal structure diagram can be as shown in Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a computer storage medium and an internal memory. The computer storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the computer storage medium. The database of the computer device is used to store data generated or obtained in the process of executing the neural network-based speech synthesis method, such as target text information and target mel spectrum. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a neural network-based speech synthesis method.

[0149] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the steps of the neural network-based speech synthesis method in the above-mentioned embodiments. Alternatively, the processor executes the computer program to implement the functions of each module / unit in the neural network-based speech synthesis system embodiment.

[0150] In an embodiment, a computer storage medium is provided, and the computer storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the neural network-based speech synthesis method in the above-mentioned embodiments. Alternatively, the computer program is executed by the processor to implement the functions of each module / unit in the neural network-based speech synthesis system embodiment.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0153] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit it. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features. The modification or replacement does not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A neural network-based speech synthesis method, characterized by, The method comprises the following steps: obtaining target text information; inputting the target text information into a speech synthesis model for processing to obtain a target mel spectrum corresponding to the target text information, wherein the speech synthesis model is an autoregressive speech synthesis model, the speech synthesis model is obtained by training a training text and a speech label, and the speech synthesis model realizes alignment between the target text information and a target speech according to a linear relationship between the training text and the speech label in a training process; the speech synthesis model comprises an encoder and an attention module, and the speech synthesis model realizes alignment between the target text information and the target speech according to a linear relationship between the training text and the speech label in a training process, which comprises: inputting the training text into the encoder to output an encoding sequence corresponding to the training text; projecting the encoding sequence corresponding to the training text and a preset attention feature into the attention module to obtain an attention weight matrix corresponding to the training text, wherein the preset attention feature is a position attention feature or a content attention feature; obtaining a training error according to the attention weight matrix corresponding to the training text and a preset loss function, adjusting network parameters of the attention module if the training error is outside a preset error threshold, and retraining the attention module according to the adjusted parameters until the training error obtained by retraining is within the preset error threshold, and the difference between the elements on the diagonal line and the elements outside the diagonal line in the attention weight matrix corresponding to the training sample obtained by recalculation is greater than a preset difference threshold, and finally obtaining a trained encoder and a trained attention module, wherein the preset loss function is obtained according to a probability mass function of the training text and the speech label and the attention weight matrix; wherein the encoder is composed of a Pre-net preprocessing network and a CBHG module, and the CBHG module is sequentially composed of a one-dimensional convolution filter group, a residual connection, a multi-layer highway network and a bidirectional gated recurrent unit GRU network; generating the target speech based on the target mel spectrum.

2. The neural network-based speech synthesis method of claim 1, wherein, The speech synthesis model further comprises a decoder, and the inputting the target text information into the speech synthesis model for processing to obtain the target mel spectrum corresponding to the target text information comprises: inputting the target text information into the trained encoder to output an encoding sequence corresponding to the target text information; projecting the encoding sequence corresponding to the target text information and a preset attention feature into the trained attention module to obtain an attention weight matrix corresponding to the target text information, and obtaining a context vector corresponding to the target text information according to the attention weight matrix corresponding to the target text information and the encoding sequence corresponding to the target text information; inputting the attention weight matrix corresponding to the target text information and the context vector corresponding to the target text information into the decoder to obtain the target mel spectrum corresponding to the target text information.

3. The neural network-based speech synthesis method of claim 1, wherein, The calculation formula of the preset loss function is as follows: ， ， ， wherein L represents the preset loss function, represents an attention weight matrix corresponding to the training text, represents a character position in the training text, represents a position of a frame in the predicted speech, represents a total number of frames of the predicted speech, represents a probability mass function of a beta-binary distribution, represents a beta function.

4. The neural network-based speech synthesis method of claim 1, wherein, The preset attention feature is a position attention feature or a content attention feature.

5. The neural network-based speech synthesis method of claim 1, wherein, The speech synthesis model is one of Tacotron, a speech synthesis model optimized based on Tacotron, DeepVoice, and a speech synthesis model optimized based on DeepVoice.

6. The neural network-based speech synthesis method according to any one of claims 1 to 4, characterized by, The target text information is obtained by: The input text data is subjected to character embedding, and a fixed-dimension vector is used to represent the input text data, and then the input text data is sequentially subjected to Pre-Net and CBHG sub-networks to obtain the target text information.

7. A neural network-based speech synthesis system for implementing the neural network-based speech synthesis method according to any one of claims 1 to 6, characterized by, The method comprises the steps of: The target text information is obtained by: The target text information is input into a speech synthesis model for processing to obtain a target mel-frequency spectrum corresponding to the target text information, wherein the speech synthesis model is an autoregressive speech synthesis model, the speech synthesis model is trained based on training text and a speech label, and the speech synthesis model realizes alignment between the target text information and a target speech based on a linear relationship between the training text and the speech label during training. The speech synthesis model comprises an encoder and an attention module, and the speech synthesis model realizes alignment between the target text information and a target speech based on a linear relationship between training text and a speech label during training, comprising: The training text is input into the encoder to output an encoding sequence corresponding to the training text; The encoding sequence corresponding to the training text and a preset attention feature are projected into the attention module to obtain an attention weight matrix corresponding to the training text, and the preset attention feature is a position attention feature or a content attention feature. According to the attention weight matrix corresponding to the training text and a preset loss function, a training error is obtained, if the training error is outside a preset error threshold, the network parameters of the attention module are adjusted, and the attention module is retrained according to the adjusted parameters until the training error obtained by retraining is within the preset error threshold, and the difference between the elements on the diagonal line position and the elements outside the diagonal line position in the attention weight matrix corresponding to the training sample obtained by re-computing is greater than a preset difference threshold, and finally the trained encoder and the trained attention module are obtained, wherein the preset loss function is obtained based on the probability mass function of the training text and the speech label and the attention weight matrix. The target speech is generated based on the target mel-frequency spectrum.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the neural network-based speech synthesis method according to any one of claims 1 to 6.

9. A computer storage medium storing a computer program, the computer program comprising instructions, which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the neural network-based speech synthesis method according to any one of claims 1 to 6.

Images