Low-resource speech synthesis system based on BERT feature and style coding

Abstract

The invention provides a low-resource speech synthesis system based on BERT characteristics and style coding, and relates to the technical field of speech synthesis, the system comprises an audio acquisition module used for acquiring to-be-synthesized audio data, and the audio data comprises a Mel spectrum and text data; the Mel spectrum style coding module is used for performing tone prediction processing according to the Mel spectrum to obtain style characteristics; the BERT text coding module is used for performing word and statement prediction processing according to the text data to obtain text features; the alignment framework module is used for performing pitch alignment prediction processing according to the Mel spectrum and the text data to obtain a pitch energy feature; and the standardized fusion module is used for performing standardized fusion processing on the style features, the text features and the audio high-energy features to obtain target synthesized audio data. According to the system, the problem that accurate human voice cannot be synthesized under the condition of low resources in an existing voice synthesis technology is solved.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a low-resource speech synthesis system based on BERT features and style coding. Background Art

[0002] In recent years, with the advancement of AI technology, speech synthesis has achieved good performance under the premise of large-scale high-quality speech data. However, in the case of low-resource data, that is, when only a small amount of user recording data is used for speech synthesis, the speech synthesis effect often has problems such as inconsistent timbre, lack of rhythm, and inaccurate pronunciation. Therefore, for the current TTS technology, it is a relatively challenging task to synthesize high-quality audio that is comparable to single-person speech synthesis in timbre and sound quality given a specified text based on a small amount of data.

[0003] However, the current low-resource personalized speech synthesis algorithm strategy mainly adopts a combination of pre-training and fine-tuning, that is, first pre-training the model on a large multi-speaker corpus dataset, and then fine-tuning the model with a small amount of data from the target speaker. Due to the small amount of data from the target speaker, current speech synthesis technology often has problems such as low timbre similarity, lack of rhythm, and inaccurate pronunciation. Summary of the invention

[0004] The present application provides a low-resource speech synthesis system based on BERT features and style coding to solve the problem that existing sound synthesis technology cannot synthesize relatively accurate human voice under low-resource conditions.

[0005] The system comprises: An audio acquisition module, wherein the audio acquisition module is configured to acquire audio data to be synthesized, wherein the audio data includes mel spectrum and text data; A mel-spectrogram style encoding module, wherein the mel-spectrogram style encoding module is configured to perform timbre prediction processing according to the mel-spectrogram to obtain style features; the timbre prediction processing includes global encoding according to the mel-spectrogram at the frame level and the time sequence corresponding to the mel-spectrogram; wherein the mel-spectrogram style encoding module includes a plurality of attention layers with stacked distribution, each of which is composed of a plurality of attention heads arranged in parallel; the mel-spectrogram style encoding module is further configured to obtain a plurality of subsequences according to the mel-spectrogram, dynamically determine one or more attention heads corresponding to each subsequence, calculate the attention weights of the subsequences according to the one or more attention heads in the attention layer, and complete global encoding according to the attention weights; A BERT text encoding module, wherein the BERT text encoding module is configured to perform word and sentence prediction processing according to the text data to obtain text features; the word and sentence prediction processing includes sentence coding prediction, word coding prediction, splicing processing according to the results of sentence coding prediction and word coding prediction, and text training according to the result after the splicing processing; An alignment framework module, wherein the alignment framework module is configured to perform pitch alignment prediction processing according to the mel spectrum and the text data to obtain a pitch energy feature; A standardization fusion module is configured to perform a standardization fusion process on the style feature, the text feature and the pitch energy feature to obtain target synthesized audio data.

[0006] Preferably, the mel spectrum style encoding module is further configured as follows: Performing frame-level attention mechanism feature prediction on the Mel spectrum to obtain the style feature; the attention mechanism feature prediction is performed in time sequence.

[0007] Preferably, the mel spectrum style encoding module includes: A spectral fully connected unit, wherein the spectral fully connected unit is configured to convert the mel spectrum into a hidden state sequence at a frame level through a built-in fully connected layer; A global feature modeling unit, wherein the global feature modeling unit is configured to capture the timing information in the hidden state sequence using a gated convolutional neural network, wherein the timing information includes the timing corresponding to each frame number of the hidden state sequence; wherein the gated convolutional neural network includes a plurality of gated convolutional layers, and the plurality of gated convolutional layers are processed by residual connection; An attention mechanism and a residual connection unit, wherein the attention mechanism and the residual connection unit are composed of a plurality of attention layers with stacked distribution, a residual connection is used between the plurality of attention layers, and each attention layer is composed of a plurality of attention heads arranged in parallel; the attention mechanism and the residual connection unit are configured to globally encode the hidden state sequence according to the temporal information to obtain the style feature; In the attention mechanism and residual connection unit, each of the attention layers includes one or more preset first attention heads and one or more second attention heads; wherein the first attention heads are all in an activated state, and the second attention heads are configured to adjust at least some of the second attention heads in the attention layer of the current level to an activated state according to the attention weights output by the attention layer of the previous level; An attention weight is calculated for the subsequence according to the first attention head and the activated second attention head.

[0008] Preferably, the attention mechanism and the residual connection unit are further configured as follows: Performing multiple attention mechanism calculations on each frame number of the hidden state sequence to obtain multiple attention mechanism calculation results corresponding to the same frame number; Calculate the average of all the attention mechanism calculation results for the same frame number to obtain an average vector; The style feature is obtained by summing up the residuals of all the average vectors of each frame.

[0009] Preferably, the BERT text encoding module is further configured as follows: Perform word coding prediction and sentence coding prediction according to the text data to obtain word semantic features and sentence semantic features respectively; The word semantic features and the sentence semantic features are concatenated to obtain the text features.

[0010] Preferably, the BERT text encoding module includes: A BERT text unit, wherein the BERT text unit is configured to perform sentence encoding prediction on the text data to obtain the sentence semantic features; the sentence encoding prediction is performed at the text sentence level; An encoder text unit, the encoder text unit is configured to perform word encoding prediction on the text data to obtain the word semantic features; the word encoding prediction is performed at the text word level; A text splicing unit is configured to splice the sentence semantic features and the word semantic features to obtain a spliced ​​text unit, and perform text training on the spliced ​​text unit to obtain the text features.

[0011] Preferably, the BERT text unit includes a plurality of transformer codes, a plurality of attention mechanism heads and a plurality of hidden layer units; The encoder text unit includes a high-level feature extraction unit and a fully connected layer.

[0012] Preferably, the alignment framework module is further configured as follows: Performing convolution encoding on the mel spectrum and the text data respectively to obtain a convolution mel spectrum and convolution text data; Performing unsupervised alignment calculation on the convolution Mel spectrum and the convolution text data to obtain pitch features and energy features; The pitch energy feature includes the pitch feature and the energy feature.

[0013] Preferably, the alignment framework module comprises: A convolutional coding unit, wherein the convolutional coding unit is configured to perform convolutional coding on the mel spectrum and the text data respectively to obtain a convolutional mel spectrum and convolutional text data; An unsupervised alignment unit is configured to perform unsupervised alignment calculation on the convolution Mel spectrum and the convolution text data according to paired L2 distances to obtain the pitch energy feature; and use a forward sum algorithm to add the alignment data obtained during the unsupervised alignment calculation.

[0014] Preferably, the standardization fusion module further includes: The style text standardization fusion unit is configured as follows: Normalizing the style features to obtain normalized data; Respectively performing scaling processing and shift processing on the normalized data to obtain scaled and shifted data; Performing full connection layer calculation on the scaled shifted data to obtain activation function and bias data; The activation function, the bias data, the text data and the pitch energy feature are concatenated to obtain synthetic audio data; A decoder unit is configured to decode the synthesized audio data to obtain the target synthesized audio data.

[0015] As can be seen from the above content, the present application provides a low-resource speech synthesis system based on BERT features and style coding, the system comprising an audio acquisition module, the audio acquisition module is configured to acquire audio data to be synthesized, the audio data comprising mel spectrum and text data; a mel spectrum style coding module, the mel spectrum style coding module is configured to perform timbre prediction processing according to the mel spectrum to obtain style features; a BERT text coding module, the BERT text coding module is configured to perform word and sentence prediction processing according to the text data to obtain text features; an alignment framework module, the alignment framework module is configured to perform pitch alignment prediction processing according to the mel spectrum and the text data to obtain pitch energy features; a standardized fusion module, the standardized fusion module is configured to perform standardized fusion processing on the style features, the text features and the pitch energy features to obtain target synthesized audio data. The present application solves the problem that the existing sound synthesis technology cannot synthesize more accurate human voices under low resource conditions through the above system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 A schematic diagram of a low-resource speech synthesis system based on BERT features and style coding for this application; Figure 2 A schematic diagram of a BERT text unit in a low-resource speech synthesis system based on BERT features and style encoding is provided for this application; Figure 3 A schematic diagram of an encoder text unit in a low-resource speech synthesis system based on BERT features and style coding in this application; Figure 4 This application provides a working principle diagram of an alignment framework module in a low-resource speech synthesis system based on BERT features and style coding. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Figure 1 A schematic diagram of a low-resource speech synthesis system based on BERT features and style coding is provided in this application.

[0020] See also Figure 1 It can be seen that this embodiment provides a low-resource speech synthesis system based on BERT features and style coding, and the system includes: The audio acquisition module 100 is configured to acquire audio data to be synthesized, wherein the audio data includes mel-spectrograms and text data. Specifically, in this embodiment, the audio acquisition module 100 is used to acquire audio data to be synthesized, wherein the audio data can be a human voice recorded in real time or a preset human voice.

[0021] Exemplarily, when a user makes a sound, the human voice is extracted through the audio acquisition module 100 and the corresponding audio data is generated; the sound recorded by the user is imported into the audio acquisition module 100, and the audio data is extracted from the imported sound through the audio acquisition module 100.

[0022] It should be noted that the audio acquisition module 100 can not only extract the audio data from the recorded sound, but also extract the audio data from the video data with sound, and obtain the audio data.

[0023] The audio data includes the Mel spectrum and text data, wherein the Mel spectrum, also known as Mel Spectrum, is an important feature extraction method in the field of speech signal processing, which maps the linear spectrum to a nonlinear spectrum based on the auditory characteristics of the human ear. Mel spectrum simulates the difference in the human ear's perception of sounds of different frequencies to more effectively analyze and process audio signals, and is widely used in multiple fields such as fundamental frequency detection, noise reduction, encoding and decoding, and special sound detection.

[0024] In sound signal processing, the original audio signal is a one-dimensional time domain signal, and it is difficult to directly analyze the changes in the frequency domain. Through methods such as Fast Fourier Transform (FFT), the time domain signal can be converted into a frequency domain signal, and then the frequency domain characteristics of the signal can be analyzed. However, although this conversion provides rich frequency domain information, it ignores the nonlinear characteristics of the human ear's perception of sound frequency.

[0025] The human ear's perception of sound is not entirely based on the linear relationship of physical frequency, but rather a complex psychoacoustic effect. Specifically, the human ear is more sensitive to low-frequency sounds and can distinguish smaller frequency changes; while for high-frequency sounds, this ability to distinguish is significantly reduced. Therefore, directly using physical frequency as the basis for analysis cannot accurately reflect the actual perception of sound by the human ear.

[0026] Mel spectrum is designed based on this principle. It uses a set of Mel filter banks based on Mel scale to map the frequency components of the audio signal to Mel scale. Mel scale is a psychoacoustic scale designed based on the frequency characteristics of human ear perception. The low frequency part is densely distributed and the high frequency part is sparsely distributed, which is more in line with the auditory characteristics of the human ear.

[0027] The system further comprises: A mel-spectrogram style encoding module 200, wherein the mel-spectrogram style encoding module 200 is configured to perform timbre prediction processing according to the mel-spectrogram to obtain style features; the timbre prediction processing includes performing global encoding according to the mel-spectrogram at the frame level and the time sequence corresponding to the mel-spectrogram; wherein the mel-spectrogram style encoding module 200 includes a plurality of attention layers with a stacked distribution, each of which is composed of a plurality of attention heads arranged in parallel; the mel-spectrogram style encoding module is further configured to obtain a plurality of subsequences according to the mel-spectrogram, dynamically determine one or more attention heads corresponding to each subsequence, calculate the attention weights of the subsequences according to the one or more attention heads in the attention layer, and complete global encoding according to the attention weights.

[0028] Specifically, in this embodiment, the mel-spectrogram style encoding module 200 is used to predict the mel-spectrogram to obtain the style feature, which can be understood as the timbre of the corresponding sound (speaker identity, rhythm), and the mel-spectrogram style encoding module 200 can make the timbre of the synthesized sound closer to the actual real person. Exemplarily, when the mel-spectrogram is extracted from the audio acquisition module 100, the mel-spectrogram is trained and learned by the mel-spectrogram style encoding module 200, so that the timbre of the generated sound is more realistic.

[0029] Furthermore, in some embodiments, the mel spectrum style encoding module 200 is further configured to: The mel spectrum is subjected to attention mechanism feature prediction at the frame level to obtain the style feature; the attention mechanism feature prediction is performed in a time sequence. Specifically, in the present embodiment, for the training and learning of timbre, the more accurate the training data is, the more accurate the timbre of the subsequently generated sound will be. Therefore, in the present embodiment, it is necessary to perform attention mechanism feature prediction at the frame level on the mel spectrum through the mel spectrum style encoding module 200 to obtain more accurate style features.

[0030] Among them, when the Mel-spectrogram style encoding module 200 performs frame-level attention mechanism feature prediction on the Mel-spectrogram, it is necessary to take into account the sequential nature of each pronunciation. Therefore, it is necessary to add a time mechanism in the training and prediction process of the Mel-spectrogram, so that each feature point in the predicted style feature has time sequence, thereby avoiding the problem of inaccurate timbre due to pronunciation timing problems.

[0031] Furthermore, in some embodiments, the mel spectrum style encoding module 200 includes: A spectrum fully connected unit 210, wherein the spectrum fully connected unit 210 is configured to convert the mel spectrum into a hidden state sequence at the frame level through a built-in fully connected layer; A global feature modeling unit 220, wherein the global feature modeling unit 220 is configured to capture the timing information in the hidden state sequence using a gated convolutional neural network, wherein the timing information includes the timing corresponding to each frame number of the hidden state sequence; wherein the gated convolutional neural network includes a plurality of gated convolutional layers, and the plurality of gated convolutional layers are processed by residual connection; An attention mechanism and residual connection unit 230, wherein the attention mechanism and residual connection unit 230 is composed of a plurality of attention layers with stacked distribution, wherein residual connections are used between the plurality of attention layers, and each attention layer is composed of a plurality of attention heads arranged in parallel; the attention mechanism and residual connection unit 230 is configured to globally encode the hidden state sequence according to the temporal information to obtain the style feature; In the attention mechanism and residual connection unit 230, each of the attention layers includes one or more preset first attention heads and one or more second attention heads; wherein the first attention heads are all in an activated state, and the second attention heads are configured to adjust at least some of the second attention heads in the attention layer of the current level to an activated state according to the attention weights output by the attention layer of the previous level; An attention weight is calculated for the subsequence according to the first attention head and the activated second attention head.

[0032] Specifically, in this embodiment, the input Mel spectrum is converted into a hidden state sequence at the frame level through the spectrum fully connected unit 210 through the fully connected layer, and the global feature modeling unit 220 uses a gated convolutional neural network (Gated CNN+residual connection) to capture the temporal information in the hidden state sequence, and finally the attention mechanism and residual connection unit 230 use a multi-head attention mechanism+residual connection to encode the global information, thereby obtaining the style feature. Among them, the multi-head attention is used at the frame level to better extract style features from shorter speech.

[0033] A projection layer is set in the above unit to complete the selection of the second attention layer in the initial stage and the selection of each second attention layer.

[0034] For the attention weights calculated by each attention layer, weighted calculation is completed through a linear layer. At the same time, the attention weights calculated by each attention layer can be used to assign weights to the original hidden state sequence, that is, when the next attention layer divides the subsequence, it is based on the hidden state sequence after weight allocation.

[0035] The above settings allow the multi-head attention mechanism to dynamically call the attention heads that are more suitable for the current sequence for attention processing (i.e., the first attention head and the second attention head) during implementation. Relatively unimportant attention heads can be inactive, which not only makes the attention calculation of the sequence more in line with the requirements, but also reduces the running load of the model, further reducing the demand for resources.

[0036] Among them, the Gated CNN is a convolutional neural network (CNN) structure widely used in deep learning. It significantly enhances the model's expressiveness and learning ability by introducing a gating mechanism. This innovative design not only improves the model's ability to handle complex tasks, but also promotes its widespread application in multiple fields such as natural language processing (NLP) and computer vision (CV). Traditional convolutional neural networks perform linear transformations on input data through convolution layers, and then apply nonlinear activation functions to extract features. The gated convolutional neural network further introduces gating units on this basis. These units, as "controllers", can dynamically adjust the weights in the convolution operation, thereby achieving more refined control over the characteristics of the input data. Specifically, the Gated CNN divides the input data into two channels: one is the linear transformation result obtained directly through the convolution kernel, and the other is the gating signal used to adjust the weights of these transformations.

[0037] In Gated CNN, the most commonly used gating units include Gated Recurrent Unit (GRU) and Long Short-Term Memory Unit (LSTM). These units can capture long-term dependencies in the data by learning the contextual information of the input data and effectively pass this information in the model. Specifically, the gating signal determines which features are retained and how to combine these features by controlling the weights of the linear transformation, thereby improving the model's ability to recognize complex data patterns.

[0038] The working mechanism of Gated CNN can be summarized as the following steps: first, the input data is linearly transformed through the convolutional layer; second, the gating unit generates a gating signal based on the contextual information of the input data; finally, these gating signals are used to adjust the weights of the linear transformation to produce the final output feature map. This mechanism makes Gated CNN perform well in processing sequence data and tasks that need to capture long-term dependencies.

[0039] Due to its advantages in capturing long-term dependencies and improving the expressiveness of models, Gated CNN has been widely used in many fields. In the field of natural language processing, Gated CNN is used for tasks such as text classification and named entity recognition, effectively improving the ability to process complex language structures. In the field of computer vision, Gated CNN is used for tasks such as image segmentation and target detection. Through fine control of image features, it achieves accurate understanding and analysis of image content.

[0040] Compared with the traditional CNN structure, Gated CNN has the following significant advantages: first, it can better capture the long-term dependencies in the input data; second, by dynamically adjusting the weights, the flexibility and expressiveness of the model are improved; third, it is suitable for processing various types of complex data, including sequence data and image data. With the continuous development of deep learning technology, Gated CNN is expected to demonstrate its unique advantages in more fields and promote further innovation and application of artificial intelligence technology.

[0041] Furthermore, in some embodiments, the attention mechanism and residual connection unit 230 is further configured as follows: Performing multiple attention mechanism calculations on each frame number of the hidden state sequence to obtain multiple attention mechanism calculation results corresponding to the same frame number; Calculate the average of all the attention mechanism calculation results for the same frame number to obtain an average vector; The style feature is obtained by summing up the residuals of all the average vectors of each frame.

[0042] Specifically, in this embodiment, the hidden state sequence includes the time dimension (number of frames) and the feature dimension. The features corresponding to each frame are calculated by multiple attention mechanisms, and what is finally obtained is the average vector of the calculation results of multiple attention mechanisms of a frame. Therefore, it is necessary to obtain the average in the time dimension and add the results of each frame number to obtain the final style feature.

[0043] The system further comprises: The BERT text encoding module 300 is configured to perform word and sentence prediction processing according to the text data to obtain text features. Specifically, in this embodiment, the BERT text encoding module 300 uses the text data to perform word and sentence prediction processing respectively, so as to obtain the best training prediction results corresponding to each word and sentence composed of several words. Compared with conventional text training techniques, this embodiment uses a combination of words and sentences for training, which further improves the model's understanding of the text, so that text content that is closer to actual real people can be generated.

[0044] Furthermore, in some embodiments, the BERT text encoding module 300 is further configured to: Perform word coding prediction and sentence coding prediction according to the text data to obtain word semantic features and sentence semantic features respectively; The word semantic features and the sentence semantic features are concatenated to obtain the text features.

[0045] Specifically, in this embodiment, each sentence and each phrase in the text data is predicted respectively by the BERT text encoding module 300, and the obtained prediction results are feature spliced ​​to obtain the text features.

[0046] Among them, BERT is a large language model based on Transformer (a large model for sequence data processing with an attention mechanism). Essentially, it learns a good feature representation for text by running a self-supervised learning method based on a massive corpus.

[0047] Wherein, the BERT text encoding module 300 includes: A BERT text unit 310, wherein the BERT text unit 310 is configured to perform sentence encoding prediction on the text data to obtain the sentence semantic features; the sentence encoding prediction is performed at the text sentence level; An encoder text unit 320, the encoder text unit 320 is configured to perform word encoding prediction on the text data to obtain the word semantic features; the word encoding prediction is performed at the text word level; The text concatenation unit 330 is configured to concatenate the sentence semantic features and the word semantic features to obtain a concatenated text unit, and perform text training on the concatenated text unit to obtain the text features.

[0048] Specifically, in this embodiment, the BERT text unit 310 and the encoder text unit 320 are respectively responsible for their own predictions, and the levels of data required for the predictions of the two are also different to a certain extent; since the BERT text unit 310 needs to perform training predictions on sentences, the BERT text unit 310 needs to perform training predictions at the text sentence level, and the encoder text unit 320 needs to perform training predictions on words, so the encoder text unit 320 needs to perform training predictions at the text word level; the text splicing unit 330 can be understood as a splicing layer in a convolutional neural network, and the text splicing unit 330 splices features of different levels, which can also be understood as features of different dimensions, that is, splicing the sentence semantic features and the word semantic features.

[0049] For example, in the TTS (text to speech) task, the function of the BERT text encoding module 300 is to act as a feature extractor after fine-tuning or fixing. We can directly use the features extracted by the BERT text unit 310 combined with the features generated in the encoder text unit 320 to represent the word embedding features for speech synthesis.

[0050] Figure 2 A schematic diagram of a BERT text unit in a low-resource speech synthesis system based on BERT features and style encoding is provided in this application.

[0051] Figure 3 A schematic diagram of an encoder text unit in a low-resource speech synthesis system based on BERT features and style coding in the present application.

[0052] See also Figure 2 and Figure 3 It can be seen that, further, in some embodiments, the BERT text unit 310 includes a number of transformer codes, a number of attention mechanism heads, and a number of hidden layer units; The encoder text unit 320 includes a high-level feature extraction unit and a fully connected layer.

[0053] Specifically, in this embodiment, the network architecture of BERT uses a multi-layer Transformer structure. Its biggest feature is that it abandons the traditional RNN (recurrent neural network) and CNN (convolutional neural network), and converts the distance between two words at any position into 1 through the Attention mechanism, effectively solving the thorny long-term dependency problem in NLP (natural language processing).

[0054] Exemplarily, this embodiment uses the Chinese RoBERTa-wwm-large-ext model, which includes 24 layers of Transformers, 16 Attention Heads, and 1024 hidden units.

[0055] The system further comprises: The alignment framework module 400 is configured to perform pitch alignment prediction processing according to the Mel-spectrogram and the text data to obtain a pitch energy feature. Specifically, in this embodiment, the alignment framework module 400 performs pitch alignment prediction processing according to the Mel-spectrogram and the text data to obtain the pitch energy feature. It can be understood that the alignment framework module 400 predicts the pitch of the sound.

[0056] Figure 4 This application provides a working principle diagram of an alignment framework module in a low-resource speech synthesis system based on BERT features and style coding.

[0057] See also Figure 4 It can be seen that the alignment framework module 400 is also configured as follows: Performing convolution encoding on the mel spectrum and the text data respectively to obtain a convolution mel spectrum and convolution text data; The convolution Mel spectrum and the convolution text data are subjected to unsupervised alignment calculation to obtain pitch features and energy features; the pitch energy features include the pitch features and the energy features.

[0058] Specifically, in this embodiment, an unsupervised alignment learning framework is used, which eliminates the need for MFA (model-free adaptive control) alignment. In the case of a small amount of data, the alignment accuracy is higher and the convergence speed is faster, which is suitable for various non-autoregressive TTS models. The pre-training + fine-tuning method is also adopted, and the alignment effect of the Alignment Framework is used to better match the rhythm of the target person.

[0059] Further, in some embodiments, the alignment framework module 400 includes: A convolutional coding unit 410, wherein the convolutional coding unit 410 is configured to perform convolutional coding on the mel spectrum and the text data respectively to obtain a convolutional mel spectrum and convolutional text data; The unsupervised alignment unit 420 is configured to perform unsupervised alignment calculation on the convolution Mel spectrum and the convolution text data according to the paired L2 distance to obtain the pitch energy feature; and use a forward sum algorithm to add the alignment data obtained in the unsupervised alignment calculation process.

[0060] Specifically, in this embodiment, the convolutional encoding unit 410 (non-autoregressive model) uses a simple 1D convolution to encode the text data and the mel spectrum, and the unsupervised alignment unit 420 uses the paired L2 distance to calculate the alignment. In order to learn these alignments, we optimize the following objectives: using the forward sum algorithm used in the hidden Markov model (HMMs), maximize the probability of a given mel spectrum. The formula is as follows: (1); in represents a specific alignment between the mel spectrum and text data, represents text input, X represents Mel spectrum, represents the set of all possible valid monotonic alignments, is the probability of aligning a particular text at time t with the mel spectrum at the corresponding time. As can be seen, this objective summarizes all possible alignments. It is proven that this objective is equivalent to minimizing the CTC (temporal class classification) loss when we restrict it to monotonic alignments.

[0061] The alignment framework module 400 eliminates the need for MFA in TTS, thereby avoiding the problem that MFA is expensive to use and is generally not easy to use for certain languages. The overall speech quality is improved after using the Alignment Framework; the Alignment Framework makes the alignment error of long utterances smaller and converges faster.

[0062] The system further comprises: The standardized fusion module 500 is configured to perform standardized fusion processing on the style features, the text features and the pitch energy features to obtain target synthesized audio data. Specifically, in this embodiment, various types of features are fused through the standardized fusion module 500 to finally obtain the target synthesized audio data.

[0063] Furthermore, in some embodiments, the standardization fusion module 500 further includes: The style text standardization fusion unit 510 is configured as follows: Normalizing the style features to obtain normalized data; Respectively performing scaling processing and shift processing on the normalized data to obtain scaled and shifted data; Performing full connection layer calculation on the scaled shifted data to obtain activation function and bias data; The activation function, the bias data, the text data and the pitch energy feature are concatenated to obtain synthetic audio data; The decoder unit 520 is configured to decode the synthesized audio data to obtain the target synthesized audio data.

[0064] Specifically, in this embodiment, in traditional practice, Style Embeddings are generally directly concatenated or added to text features or decoder output. In this embodiment, a Layer Norm method is proposed, which receives the style vector Style Embeddings to predict the gain (activation function) and bias (bias data) of the input feature, and then performs normalization to obtain the output. The formula is as follows: (2) in , ; (3); in, is the input feature, for The dimension of is the value after mean variance normalization, is the style vector, μ is the average value of all sample feature vectors, h i is the eigenvector of the i-th sample, σ represents the standard deviation of the eigenvector, and SALN in the formula represents and function.

[0065] Unlike the normal LayerNorm, and The normalized input features can be scaled and shifted, and this step makes the style vector more in line with expectations. Style-Adaptive Layer Norm (SALN) is applied in the encoder text unit 320 and the decoder unit 520 of the Decoder (i.e., the Layer Norm is replaced with SALN), where bias and gain are obtained by the style vector Style Embeddings through a fully connected layer.

[0066] This embodiment has the following advantages: Low-resource speech synthesis technology based on BERT features has significant advantages. It uses non-autoregressive speech synthesis methods as the basis. While ensuring the real-time performance of speech synthesis, it also adds word semantic features and sentence semantic features to train the acoustic model, making the speech synthesis more accurate and rhythmic. The style features of the audio are extracted using the Mel-spectrogram style coding module, and are added to the main module of the speech synthesis algorithm using the Style-Adaptive Layer Norm method. This can retain the timbre of the speaker to be synthesized to the greatest extent. The alignment framework based on CTC loss is used instead of MFA to participate in the model training of speech synthesis and improve the accuracy of alignment.

Claims

1. A low-resource speech synthesis system based on BERT features and style coding, characterized in that: The system comprises: An audio acquisition module (100), the audio acquisition module (100) being configured to acquire audio data to be synthesized, the audio data comprising mel spectrum and text data; A mel-spectrogram style encoding module (200), wherein the mel-spectrogram style encoding module (200) is configured to perform timbre prediction processing according to the mel-spectrogram to obtain style features; the timbre prediction processing includes performing global encoding according to the mel-spectrogram at the frame level and the time sequence corresponding to the mel-spectrogram; wherein the mel-spectrogram style encoding module (200) includes a plurality of attention layers with stacked distribution, each of which is composed of a plurality of attention heads arranged in parallel; the mel-spectrogram style encoding module is further configured to obtain a plurality of subsequences according to the mel-spectrogram, dynamically determine one or more attention heads corresponding to each subsequence, calculate attention weights for the subsequences according to the one or more attention heads in the attention layer, and complete global encoding according to the attention weights; A BERT text encoding module (300), wherein the BERT text encoding module (300) is configured to perform word and sentence prediction processing according to the text data to obtain text features; the word and sentence prediction processing includes sentence coding prediction, word coding prediction, splicing processing according to the results of the sentence coding prediction and word coding prediction, and text training according to the result after the splicing processing; An alignment framework module (400), wherein the alignment framework module (400) is configured to perform pitch alignment prediction processing according to the mel spectrum and the text data to obtain a pitch energy feature; A standardization fusion module (500) is configured to perform a standardization fusion process on the style feature, the text feature and the pitch energy feature to obtain target synthesized audio data.

2. A low-resource speech synthesis system based on BERT features and style coding according to claim 1, characterized in that: The mel spectrum style encoding module (200) is further configured to: Performing frame-level attention mechanism feature prediction on the Mel spectrum to obtain the style feature; the attention mechanism feature prediction is performed in time sequence.

3. A low-resource speech synthesis system based on BERT features and style coding according to claim 2, characterized in that: The mel spectrum style encoding module (200) comprises: A spectral fully connected unit (210), wherein the spectral fully connected unit (210) is configured to convert the mel-spectrogram into a hidden state sequence at a frame level through a built-in fully connected layer; A global feature modeling unit (220), the global feature modeling unit (220) being configured to capture the timing information in the hidden state sequence using a gated convolutional neural network, the timing information comprising the timing corresponding to each frame number of the hidden state sequence; wherein the gated convolutional neural network comprises a plurality of gated convolutional layers, and the plurality of gated convolutional layers are processed by a residual connection method; An attention mechanism and a residual connection unit (230), wherein the attention mechanism and the residual connection unit (230) are composed of a plurality of attention layers with stacked distribution, a residual connection is used between the plurality of attention layers, and each attention layer is composed of a plurality of attention heads arranged in parallel; the attention mechanism and the residual connection unit (230) are configured to globally encode the hidden state sequence according to the temporal information to obtain the style feature; In the attention mechanism and residual connection unit (230), each of the attention layers includes one or more preset first attention heads and one or more second attention heads; wherein the first attention heads are all in an activated state, and the second attention heads are configured to adjust at least part of the second attention heads in the attention layer of the current level to an activated state according to the attention weight output by the attention layer of the previous level; An attention weight is calculated for the subsequence according to the first attention head and the activated second attention head.

4. A low-resource speech synthesis system based on BERT features and style coding according to claim 3, characterized in that: The attention mechanism and residual connection unit (230) is also configured as follows: Performing multiple attention mechanism calculations on each frame number of the hidden state sequence to obtain multiple attention mechanism calculation results corresponding to the same frame number; Calculate the average of all the attention mechanism calculation results for the same frame number to obtain an average vector; The style feature is obtained by summing up the residuals of all the average vectors of each frame.

5. A low-resource speech synthesis system based on BERT features and style coding according to claim 1, characterized in that: The BERT text encoding module (300) is further configured to: Perform word coding prediction and sentence coding prediction according to the text data to obtain word semantic features and sentence semantic features respectively; The word semantic features and the sentence semantic features are concatenated to obtain the text features.

6. A low-resource speech synthesis system based on BERT features and style coding according to claim 5, characterized in that: The BERT text encoding module (300) includes: A BERT text unit (310), wherein the BERT text unit (310) is configured to perform sentence encoding prediction on the text data to obtain the sentence semantic features; the sentence encoding prediction is performed at the text sentence level; An encoder text unit (320), the encoder text unit (320) being configured to perform word encoding prediction on the text data to obtain the word semantic features; the word encoding prediction is performed at a text word level; A text concatenation unit (330) is configured to concatenate the sentence semantic features and the word semantic features to obtain a concatenated text unit, and to perform text training on the concatenated text unit to obtain the text features.

7. A low-resource speech synthesis system based on BERT features and style coding according to claim 6, characterized in that: The BERT text unit (310) includes a plurality of transformer codes, a plurality of attention mechanism heads and a plurality of hidden layer units; The encoder text unit (320) includes a high-level feature extraction unit and a fully connected layer.

8. A low-resource speech synthesis system based on BERT features and style coding according to claim 1, characterized in that: The alignment framework module (400) is further configured to: Performing convolution encoding on the mel spectrum and the text data respectively to obtain a convolution mel spectrum and convolution text data; Performing unsupervised alignment calculation on the convolution Mel spectrum and the convolution text data to obtain pitch features and energy features; The pitch energy feature includes the pitch feature and the energy feature.

9. A low-resource speech synthesis system based on BERT features and style coding according to claim 8, characterized in that: The alignment framework module (400) comprises: A convolutional coding unit (410), wherein the convolutional coding unit (410) is configured to perform convolutional coding on the mel spectrum and the text data respectively to obtain a convolutional mel spectrum and convolutional text data; An unsupervised alignment unit (420) is configured to perform unsupervised alignment calculation on the convolution Mel spectrum and the convolution text data according to paired L2 distances to obtain the pitch energy feature; and use a forward sum algorithm to add the alignment data obtained during the unsupervised alignment calculation process.

10. A low-resource speech synthesis system based on BERT features and style coding according to claim 1, characterized in that: The standardization fusion module (500) further includes: The style text standardization fusion unit (510) is configured to: Normalizing the style features to obtain normalized data; Respectively performing scaling processing and shift processing on the normalized data to obtain scaled and shifted data; Performing full connection layer calculation on the scaled shifted data to obtain activation function and bias data; The activation function, the bias data, the text data and the pitch energy feature are concatenated to obtain synthetic audio data; A decoder unit (520), wherein the decoder unit (520) is configured to perform decoding processing on the synthesized audio data to obtain the target synthesized audio data.

Images