A low-resource speech synthesis system based on BERT features and style encoding

Abstract

The application provides a low-resource speech synthesis system based on BERT features and style encoding, and relates to the technical field of speech synthesis. The system comprises an audio acquisition module for acquiring audio data to be synthesized, the audio data comprising a mel spectrum and text data; a mel spectrum style encoding module for performing timbre prediction processing according to the mel spectrum to obtain style features; a BERT text encoding module for performing word and sentence prediction processing according to the text data to obtain text features; an alignment framework module for performing pitch alignment prediction processing according to the mel spectrum and the text data to obtain pitch energy features; and a standardization fusion module for performing standardization fusion processing on the style features, the text features and the pitch energy features to obtain target synthesized audio data. The application solves the problem that existing sound synthesis technology cannot synthesize relatively accurate human voices in a low-resource situation.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a low-resource speech synthesis system based on BERT feature and style coding. Background Technology

[0002] In recent years, with the advancement of AI technology, current TTS (Text-to-Speech) models have shown promising performance under conditions of large-scale, high-quality speech data. However, in situations with low-resource data, i.e., when using only a small amount of user recordings for speech synthesis, the results often suffer from issues such as inconsistent timbre, lack of rhythm, and inaccurate pronunciation. Therefore, for current TTS technology, synthesizing high-quality audio with comparable timbre and sound quality to single-person speech synthesis, given a limited amount of data and a specified text, remains a challenging task.

[0003] However, current low-resource personalized speech synthesis algorithms mainly employ a combination of pre-training and fine-tuning. This involves pre-training the model on a large multi-speaker corpus and then fine-tuning it using a small amount of data from the target speaker. Due to the limited amount of data from the target speaker, current speech synthesis techniques often suffer from low timbre similarity, lack of prosody, and inaccurate pronunciation. Summary of the Invention

[0004] This application provides a low-resource speech synthesis system based on BERT feature and style coding to solve the problem that existing voice synthesis technologies cannot synthesize relatively accurate human voices under low resource conditions.

[0005] The system includes:

[0006] An audio acquisition module is configured to acquire audio data to be synthesized, the audio data including Mel spectrum and text data;

[0007] A Mel spectrum style encoding module is configured to perform timbre prediction processing based on the Mel spectrum to obtain style features. The timbre prediction processing includes global encoding based on the frame-level Mel spectrum and the corresponding temporal sequence. The Mel spectrum style encoding module includes multiple attention layers arranged in a stacked manner, each attention layer consisting of multiple parallel attention heads. The module is further configured to divide the Mel spectrum into multiple sub-sequences, dynamically determine one or more attention heads corresponding to each sub-sequence, calculate attention weights for each sub-sequence based on the one or more attention heads in the attention layers, and complete global encoding based on the attention weights.

[0008] The BERT text encoding module is configured to perform word and sentence prediction processing on the text data to obtain text features; the word and sentence prediction processing includes sentence encoding prediction, word encoding prediction, concatenation processing based on the results of sentence encoding prediction and word encoding prediction, and text training based on the concatenation processing results;

[0009] An alignment framework module is configured to perform pitch alignment prediction processing based on the Mel spectrum and the text data to obtain pitch energy features;

[0010] A standardization fusion module is configured to perform standardization fusion processing on the style features, the text features, and the pitch energy features to obtain target synthesized audio data.

[0011] Preferably, the Mel-spectrum style coding module is further configured as follows:

[0012] The style features are obtained by performing frame-level attention mechanism feature prediction on the Mel spectrum; the attention mechanism feature prediction is performed in a temporal sequence.

[0013] Preferably, the Mel spectrum style encoding module includes:

[0014] A fully connected spectral unit is configured to convert the Mel spectrum into a sequence of hidden states at the frame level through a built-in fully connected layer.

[0015] A global feature modeling unit is configured to capture temporal information in the hidden state sequence using a gated convolutional neural network. The temporal information includes the temporal sequence corresponding to each frame number of the hidden state sequence. The gated convolutional neural network includes multiple gated convolutional layers, which are processed by residual connections.

[0016] The attention mechanism and residual connection unit are composed of multiple attention layers arranged in a stacked manner, with residual connections between the multiple attention layers. Each attention layer is composed of multiple attention heads set in parallel. The attention mechanism and residual connection unit are configured to globally encode the hidden state sequence according to the temporal information to obtain the style features.

[0017] In the attention mechanism and residual connection unit, each attention layer includes one or more preset first attention heads and one or more second attention heads; wherein, the first attention heads are all in an active state, and the second attention heads are configured to adjust at least some of the second attention heads in the current level attention layer to an active state according to the attention weight output by the attention layer of the previous level.

[0018] Attention weights are calculated for the subsequence based on the first attention head and the second attention head that is in an active state.

[0019] Preferably, the attention mechanism and residual connection unit are further configured as follows:

[0020] For each frame number of the hidden state sequence, the attention mechanism is calculated multiple times to obtain multiple attention mechanism calculation results corresponding to the same frame number;

[0021] The average of all attention mechanism calculations for the same frame number is calculated to obtain the average vector.

[0022] The style features are obtained by summing the residuals of all the average vectors for each frame.

[0023] Preferably, the BERT text encoding module is further configured as follows:

[0024] Based on the text data, word encoding prediction and sentence encoding prediction are performed respectively to obtain word semantic features and sentence semantic features respectively;

[0025] The text features are obtained by concatenating the semantic features of the words and the semantic features of the sentences.

[0026] Preferably, the BERT text encoding module includes:

[0027] A BERT text unit is configured to perform sentence encoding prediction on the text data to obtain the sentence semantic features; the sentence encoding prediction is performed at the text sentence level.

[0028] An encoder text unit is configured to perform word encoding prediction on the text data to obtain the word semantic features; the word encoding prediction is performed at the text word level.

[0029] A text concatenation unit is configured to concatenate the sentence semantic features and the word semantic features to obtain a concatenated text unit, and to perform text training based on the concatenated text unit to obtain the text features.

[0030] Preferably, the BERT text unit includes several transformer codes, several attention mechanism heads, and several hidden layer units;

[0031] The encoder text unit includes a high-level feature extraction unit and a fully connected layer.

[0032] Preferably, the alignment frame module is further configured to:

[0033] The Mel spectrum and the text data are convolutionally encoded to obtain the convolutional Mel spectrum and the convolutional text data.

[0034] Unsupervised alignment calculations are performed on the convolutional Mel spectrum and the convolutional text data to obtain pitch features and energy features;

[0035] The pitch energy feature includes the pitch feature and the energy feature.

[0036] Preferably, the alignment frame module includes:

[0037] A convolutional coding unit is configured to perform convolutional coding on the Mel spectrum and the text data respectively to obtain convolutional Mel spectrum and convolutional text data;

[0038] An unsupervised alignment unit is configured to perform unsupervised alignment calculations on the convolutional Mel spectrum and the convolutional text data according to pairwise L2 distances to obtain the pitch energy features; and to sum the alignment data obtained during the unsupervised alignment calculation process using a forward summation algorithm.

[0039] Preferably, the standardized fusion module further includes:

[0040] A style text standardization and fusion unit, wherein the style text standardization and fusion unit is configured as follows:

[0041] The style features are then normalized to obtain normalized data;

[0042] The normalized data is scaled and shifted to obtain scaled and shifted data.

[0043] The scaling-shifted data is processed by a fully connected layer to obtain the activation function and the biased data.

[0044] The activation function, the paranoia data, the text data, and the pitch energy features are concatenated to obtain synthetic audio data.

[0045] A decoder unit is configured to decode the synthesized audio data to obtain the target synthesized audio data.

[0046] As described above, this application provides a low-resource speech synthesis system based on BERT features and style encoding. The system includes an audio acquisition module configured to acquire audio data to be synthesized, the audio data including Mel spectrograms and text data; a Mel spectrogram style encoding module configured to perform timbre prediction processing based on the Mel spectrograms to obtain style features; a BERT text encoding module configured to perform word and sentence prediction processing based on the text data to obtain text features; an alignment framework module configured to perform pitch alignment prediction processing based on the Mel spectrograms and the text data to obtain pitch energy features; and a normalization fusion module configured to perform normalization fusion processing on the style features, text features, and pitch energy features to obtain target synthesized audio data. This application solves the problem that existing voice synthesis technologies cannot synthesize accurate human voices under low-resource conditions through the above system. Attached Figure Description

[0047] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of a low-resource speech synthesis system based on BERT feature and style coding according to this application;

[0049] Figure 2 This is a schematic diagram of a BERT text unit in a low-resource speech synthesis system based on BERT feature and style coding according to this application;

[0050] Figure 3 This is a schematic diagram of the encoder text unit in a low-resource speech synthesis system based on BERT feature and style coding according to this application;

[0051] Figure 4 This is a schematic diagram illustrating the working principle of the alignment framework module in a low-resource speech synthesis system based on BERT feature and style coding, as described in this application. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Figure 1 This is a schematic diagram of a low-resource speech synthesis system based on BERT feature and style coding according to this application.

[0054] See Figure 1 As can be seen, this embodiment provides a low-resource speech synthesis system based on BERT feature and style coding, the system comprising:

[0055] An audio acquisition module 100 is configured to acquire audio data to be synthesized. The audio data includes melodic spectrum and text data. Specifically, in this embodiment, the audio acquisition module 100 is used to acquire the audio data to be synthesized. The audio data can be real-time recorded human voice or preset human voice.

[0056] For example, when a user makes a sound, the audio acquisition module 100 extracts the voice and generates the corresponding audio data; the user's recorded voice is imported into the audio acquisition module 100, and the audio acquisition module 100 extracts the audio data from the imported voice.

[0057] It should be noted that the audio acquisition module 100 can not only extract the audio data from the recorded sound, but also extract it from video data containing sound to obtain the audio data.

[0058] The audio data includes the Mel spectrum and text data. The Mel spectrum, also known as the Mel Spectrum, is an important feature extraction method in the field of speech signal processing. It maps a linear spectrum onto a nonlinear spectrum based on the characteristics of human hearing. By simulating the differences in human hearing perception of different frequencies, the Mel spectrum enables more effective analysis and processing of audio signals and is widely used in fields such as fundamental frequency detection, noise reduction, encoding / decoding, and special sound detection.

[0059] In audio signal processing, the original audio signal is a one-dimensional time-domain signal, and direct analysis makes it difficult to intuitively perceive changes in the frequency domain. Methods such as the Fast Fourier Transform (FFT) can convert the time-domain signal into a frequency-domain signal, allowing for the analysis of its frequency characteristics. However, while this conversion provides rich frequency-domain information, it neglects the non-linear nature of human auditory perception of sound frequencies.

[0060] Human sound perception is not entirely based on a linear relationship between physical frequencies, but rather involves a complex psychoacoustic effect. Specifically, the human ear is more sensitive to low-frequency sounds and can distinguish smaller frequency changes; however, this ability to distinguish high-frequency sounds is significantly reduced. Therefore, directly using physical frequencies as the basis for analysis cannot accurately reflect the actual way the human ear perceives sound.

[0061] Mel-scale is designed based on this principle. It uses a set of Mel-scale-based filter banks to map the frequency components of the audio signal onto the Mel scale. The Mel scale is a psychoacoustic scale designed according to the frequency characteristics perceived by the human ear, with a dense distribution in the low-frequency part and a sparse distribution in the high-frequency part, thus better matching the auditory characteristics of the human ear.

[0062] The system also includes:

[0063] A Mel spectrum style encoding module 200 is configured to perform timbre prediction processing based on the Mel spectrum to obtain style features. The timbre prediction processing includes global encoding based on the frame-level Mel spectrum and the corresponding temporal sequence. The Mel spectrum style encoding module 200 includes multiple attention layers arranged in a stacked manner, each attention layer consisting of multiple parallel attention heads. The module is further configured to divide the Mel spectrum into multiple sub-sequences, dynamically determine one or more attention heads corresponding to each sub-sequence, calculate attention weights for each sub-sequence based on the one or more attention heads in the attention layers, and complete global encoding based on the attention weights.

[0064] Specifically, in this embodiment, the Mel spectrum style encoding module 200 is used to predict the Mel spectrum to obtain the style features. The style features can be understood as the timbre of the corresponding sound (speaker identity, rhythm). The Mel spectrum style encoding module 200 can make the timbre of the synthesized sound closer to that of a real person. For example, when the Mel spectrum is extracted from the audio acquisition module 100, the Mel spectrum is trained and learned by the Mel spectrum through the Mel spectrum style encoding module 200 to make the timbre of the generated sound more realistic.

[0065] Furthermore, in some embodiments, the Mel spectral style encoding module 200 is also configured to:

[0066] The style features are obtained by performing frame-level attention mechanism feature prediction on the Mel spectrum. The attention mechanism feature prediction is performed in sequence. Specifically, in this embodiment, for timbre training, the more accurate the training data, the more accurate the timbre of the subsequently generated sound. Therefore, in this embodiment, the Mel spectrum style encoding module 200 needs to perform frame-level attention mechanism feature prediction on the Mel spectrum to obtain more accurate style features.

[0067] When the Mel spectrum style encoding module 200 performs frame-level attention mechanism feature prediction on the Mel spectrum, it needs to consider the sequentiality of each pronunciation. Therefore, a time mechanism needs to be added during the training and prediction process of the Mel spectrum so that each feature point in the predicted style features has temporal sequence, thereby avoiding the problem of inaccurate timbre caused by pronunciation timing issues.

[0068] Furthermore, in some embodiments, the Mel spectral style coding module 200 includes:

[0069] A fully connected spectral unit 210 is configured to convert the Mel spectrum into a frame-level hidden state sequence through a built-in fully connected layer.

[0070] A global feature modeling unit 220 is configured to capture temporal information in the hidden state sequence using a gated convolutional neural network. The temporal information includes the temporal sequence corresponding to each frame number of the hidden state sequence. The gated convolutional neural network includes multiple gated convolutional layers, which are processed by residual connections.

[0071] The attention mechanism and residual connection unit 230 are composed of multiple attention layers arranged in a stacked manner, with residual connections between the multiple attention layers. Each attention layer is composed of multiple attention heads set in parallel. The attention mechanism and residual connection unit 230 are configured to globally encode the hidden state sequence according to the temporal information to obtain the style features.

[0072] In the attention mechanism and residual connection unit 230, each attention layer includes one or more preset first attention heads and one or more second attention heads; wherein, the first attention heads are all in an active state, and the second attention heads are configured to adjust at least some of the second attention heads in the current level attention layer to an active state according to the attention weight output by the attention layer of the previous level.

[0073] Attention weights are calculated for the subsequence based on the first attention head and the second attention head that is in an active state.

[0074] Specifically, in this embodiment, the input Mel spectrum is converted into a frame-level hidden state sequence through a fully connected layer by the fully connected spectral unit 210. The global feature modeling unit 220 uses a gated convolutional neural network (Gated CNN + residual connections) to capture the temporal information in the hidden state sequence. Finally, the global information is encoded using a multi-head attention mechanism + residual connections by the attention mechanism and the residual connection unit 230, thereby obtaining the style features. Multi-head attention is used at the frame level to better extract style features from shorter speech segments.

[0075] The above unit includes a projection layer for selecting the second attention layer in the initial stage, as well as for selecting the second attention layer in each subsequent stage.

[0076] The attention weights calculated by each attention layer are weighted by a linear layer. At the same time, the attention weights calculated by each attention layer can be used to allocate weights to the original hidden state sequence. That is, when the next attention layer divides the subsequence, it is based on the hidden state sequence after weight allocation.

[0077] The above settings allow the multi-head attention mechanism to dynamically call the attention head that is more suitable for the current sequence (i.e., the first attention head and the second attention head) for attention processing during implementation. For relatively unimportant attention heads, they can be kept inactive. This not only makes the attention calculation of the sequence more in line with the requirements, but also reduces the running load of the model and further reduces the demand for resources.

[0078] Gated CNN, a widely used convolutional neural network (CNN) architecture in deep learning, significantly enhances the model's expressive and learning capabilities by introducing a gating mechanism. This innovative design not only improves the model's ability to handle complex tasks but also promotes its widespread application in multiple fields such as natural language processing (NLP) and computer vision (CV). Traditional convolutional neural networks perform linear transformations on input data through convolutional layers and then apply non-linear activation functions to extract features. Gated CNN further introduces gating units, which act as "controllers" to dynamically adjust the weights in the convolution operation, thereby achieving finer control over the features of the input data. Specifically, Gated CNN divides the input data into two channels: one is the linear transformation result obtained directly through the convolution kernel, and the other is the gating signal used to adjust the weights of these transformations.

[0079] In Gated CNNs, the most commonly used gating units include Gated Recurrent Units (GRUs) and Long Short-Term Memory Units (LSTMs). These units learn contextual information from the input data, enabling them to capture long-term dependencies in the data and effectively transmit this information within the model. Specifically, gating signals control the weights of linear transformations, determining which features are preserved and how these features are combined, thereby improving the model's ability to recognize complex data patterns.

[0080] The working mechanism of Gated CNN can be summarized in the following steps: First, the input data undergoes a linear transformation through convolutional layers; second, gating units generate gating signals based on the contextual information of the input data; finally, these gating signals are used to adjust the weights of the linear transformation, producing the final output feature map. This mechanism makes Gated CNN perform exceptionally well when processing sequential data and tasks that require capturing long-term dependencies.

[0081] Due to its advantages in capturing long-term dependencies and improving model expressive power, Gated CNN has been widely used in multiple fields. In natural language processing, Gated CNN is used for tasks such as text classification and named entity recognition, effectively improving the ability to handle complex language structures. In computer vision, Gated CNN is used for tasks such as image segmentation and object detection, achieving accurate understanding and analysis of image content through fine-grained control of image features.

[0082] Compared to traditional CNN architectures, Gated CNNs offer several significant advantages: first, they are better able to capture long-term dependencies in the input data; second, by dynamically adjusting weights, they enhance the model's flexibility and expressive power; and third, they are suitable for processing various types of complex data, including sequence data and image data. With the continuous development of deep learning technology, Gated CNNs are expected to demonstrate their unique advantages in more fields, driving further innovation and application of artificial intelligence technology.

[0083] Furthermore, in some embodiments, the attention mechanism and residual connection unit 230 are also configured as follows:

[0084] For each frame number of the hidden state sequence, the attention mechanism is calculated multiple times to obtain multiple attention mechanism calculation results corresponding to the same frame number;

[0085] The average of all attention mechanism calculations for the same frame number is calculated to obtain the average vector.

[0086] The style features are obtained by summing the residuals of all the average vectors for each frame.

[0087] Specifically, in this embodiment, the hidden state sequence includes a time dimension (number of frames) and a feature dimension. The features corresponding to each frame are calculated by multiple attention mechanisms. The final result is the average vector of the calculation results of multiple attention mechanisms for a frame. Therefore, it is necessary to take the average in the time dimension and sum the results of each frame to obtain the final style feature.

[0088] The system also includes:

[0089] The BERT text encoding module 300 is configured to perform word and sentence prediction processing based on the text data to obtain text features. Specifically, in this embodiment, the BERT text encoding module 300 uses the text data to perform word and sentence prediction processing respectively, thereby obtaining the best training prediction results for each word and sentences composed of several words. Compared with conventional text training techniques, this embodiment uses a combination of words and sentences for training, further improving the model's understanding of text, thereby generating text content that is closer to real people.

[0090] Furthermore, in some embodiments, the BERT text encoding module 300 is also configured to:

[0091] Based on the text data, word encoding prediction and sentence encoding prediction are performed respectively to obtain word semantic features and sentence semantic features respectively;

[0092] The text features are obtained by concatenating the semantic features of the words and the semantic features of the sentences.

[0093] Specifically, in this embodiment, the BERT text encoding module 300 predicts each statement and sentence in the text data, and the prediction results are concatenated to obtain the text features.

[0094] BERT is a large-scale language model based on Transformer (a large-scale model for sequence data processing with attention mechanisms). Essentially, it learns a good feature representation for text by running a self-supervised learning method on a massive corpus.

[0095] The BERT text encoding module 300 includes:

[0096] BERT text unit 310 is configured to perform sentence encoding prediction on the text data to obtain the sentence semantic features; the sentence encoding prediction is performed at the text sentence level.

[0097] An encoder text unit 320 is configured to perform word encoding prediction on the text data to obtain the word semantic features; the word encoding prediction is performed at the text word level.

[0098] The text splicing unit 330 is configured to splice the sentence semantic features and the word semantic features to obtain a spliced ​​text unit, and to perform text training based on the spliced ​​text unit to obtain the text features.

[0099] Specifically, in this embodiment, the BERT text unit 310 and the encoder text unit 320 are each responsible for their respective predictions, and the levels of data required for their predictions also differ to some extent. Since the BERT text unit 310 needs to train and predict sentences, it needs to train and predict at the sentence level. The encoder text unit 320 needs to train and predict words, so it needs to train and predict at the word level. The text splicing unit 330 can be understood as a splicing layer in a convolutional neural network. Through the text splicing unit 330, features at different levels, or in other words, at different dimensions, are spliced ​​together, that is, the semantic features of the sentences and the semantic features of the words are spliced ​​together.

[0100] For example, in a TTS (text-to-speech) task, the BERT text encoding module 300 functions as a feature extractor after fine-tuning or fixing. We can directly use the features extracted by the BERT text unit 310 combined with the features generated in the encoder text unit 320 to represent the word embedding features for speech synthesis.

[0101] Figure 2 This is a schematic diagram of a BERT text unit in a low-resource speech synthesis system based on BERT feature and style coding, as described in this application.

[0102] Figure 3 This is a schematic diagram of the encoder text unit in a low-resource speech synthesis system based on BERT feature and style coding according to this application.

[0103] See Figure 2 and Figure 3 It is understood that, further, in some embodiments, the BERT text unit 310 includes several transformer codes, several attention mechanism heads, and several hidden layer units;

[0104] The encoder text unit 320 includes a high-level feature extraction unit and a fully connected layer.

[0105] Specifically, in this embodiment, BERT's network architecture uses a multi-layer Transformer structure. Its biggest feature is that it abandons the traditional RNN (Recurrent Neural Network) and CNN (Convolutional Neural Network). It uses the Attention mechanism to convert the distance between two words at any position into 1, which effectively solves the tricky long-term dependency problem in NLP (Natural Language Processing).

[0106] For example, this embodiment uses the Chinese RoBERTa-wwm-large-ext model, which contains 24 Transformers, 16 Attention Heads, and 1024 hidden units.

[0107] The system also includes:

[0108] Alignment frame module 400 is configured to perform pitch alignment prediction processing based on the Mel spectrum and the text data to obtain pitch energy features. Specifically, in this embodiment, the alignment frame module 400 performs pitch alignment prediction processing based on the Mel spectrum and the text data to obtain the pitch energy features. It can be understood that the alignment frame module 400 predicts the pitch of the sound.

[0109] Figure 4 This is a schematic diagram illustrating the working principle of the alignment framework module in a low-resource speech synthesis system based on BERT feature and style coding, as described in this application.

[0110] See Figure 4 It can be seen that the alignment frame module 400 is further configured as follows:

[0111] The Mel spectrum and the text data are convolutionally encoded to obtain the convolutional Mel spectrum and the convolutional text data.

[0112] Unsupervised alignment calculations are performed on the convolutional Mel spectrum and the convolutional text data to obtain pitch features and energy features; the pitch and energy features include the pitch features and the energy features.

[0113] Specifically, in this embodiment, an unsupervised alignment learning framework is adopted. This framework eliminates the need for MFA (Model-Free Adaptive Control) alignment, resulting in higher alignment accuracy and faster convergence speed when dealing with limited data. It is applicable to various non-autoregressive TTS models. Similarly, a pre-training + fine-tuning approach is used, and the alignment effect of the Alignment Framework better matches the rhythm and intonation of the target audience.

[0114] Furthermore, in some embodiments, the alignment frame module 400 includes:

[0115] Convolutional coding unit 410 is configured to perform convolutional coding on the Mel spectrum and the text data respectively to obtain convolutional Mel spectrum and convolutional text data;

[0116] An unsupervised alignment unit 420 is configured to perform unsupervised alignment calculations on the convolutional Mel spectrum and the convolutional text data according to the pairwise L2 distance to obtain the pitch energy features; and to sum the alignment data obtained in the unsupervised alignment calculation process using a forward summation algorithm.

[0117] Specifically, in this embodiment, the convolutional encoding unit 410 (non-autoregressive model) encodes the text data and the Mel spectrum using simple 1D convolutions, and the unsupervised alignment unit 420 calculates the alignment using pairwise L2 distances. To learn these alignments, we optimize the following objective: maximizing the probability of a given Mel spectrum using the forward summation algorithm used in Hidden Markov Models (HMMs). The formula is shown below:

[0118] (1);

[0119] in This indicates a specific alignment between the mel spectrum and the text data. This represents text input, and X represents Mel spectrum. Represents the set of all possible valid monotonic alignments. This represents the probability that a specific text will align with the Mel spectrum at time t. As can be seen, this objective summarizes all possible alignments. It has been proven that when we restrict it to monotonic alignments, this objective is equivalent to minimizing the CTC (Time-Series Classification) loss.

[0120] The alignment framework module 400 eliminates the need for an MFA in TTS, thus avoiding the high cost of MFAs and their often unsuitability for certain languages. Using the Alignment Framework improves overall speech quality; it also reduces alignment errors and accelerates convergence in long utterances.

[0121] The system also includes:

[0122] The standardization fusion module 500 is configured to perform standardization fusion processing on the style features, the text features, and the pitch energy features to obtain target synthesized audio data. Specifically, in this embodiment, the standardization fusion module 500 fuses various features to finally obtain the target synthesized audio data.

[0123] Furthermore, in some embodiments, the standardized fusion module 500 further includes:

[0124] Style text standardization fusion unit 510, wherein the style text standardization fusion unit 510 is configured as follows:

[0125] The style features are then normalized to obtain normalized data;

[0126] The normalized data is scaled and shifted to obtain scaled and shifted data.

[0127] The scaling-shifted data is processed by a fully connected layer to obtain the activation function and the biased data.

[0128] The activation function, the paranoia data, the text data, and the pitch energy features are concatenated to obtain synthetic audio data.

[0129] Decoder unit 520 is configured to decode the synthesized audio data to obtain the target synthesized audio data.

[0130] Specifically, in this embodiment, in traditional approaches, Style Embeddings are typically directly concatenated or added to text features or the decoder's output. This embodiment proposes a Layer Norm method, which takes Style Embeddings as input vectors to predict the gain (activation function) and bias (biased data) of the input features, and then performs standardization to obtain the output, as shown in the following formula:

[0131] (2)

[0132] in , ;

[0133] (3);

[0134] in, For input features, for Dimensions This is the value after normalizing the mean and variance. Let h be the style vector, μ be the average of all sample feature vectors, and h be the style vector. i Let be the feature vector of the i-th sample, and σ represent the standard deviation of the feature vector. SALN in the formula represents a feature vector based on... and The function.

[0135] Unlike a regular LayerNorm, this one... and The normalized input features can be scaled and shifted to make the style vector more consistent with expectations. Style-Adaptive Layer Norm (SALN) (i.e., replacing Layer Norm with SALN) is applied in the encoder text unit 320 and the decoder unit 520 of the decoder, where bias and gain are obtained by passing the style vectors Style Embeddings through a fully connected layer.

[0136] This embodiment has the following advantages:

[0137] Low-resource speech synthesis technology based on BERT features has significant advantages. It uses a non-autoregressive speech synthesis method as its foundation, ensuring real-time speech synthesis while incorporating word and sentence semantic features for acoustic model training. This results in more accurate and rhythmic speech synthesis. A Mel-spectral style coding module is used to extract audio style features, which are then added to the main module of the speech synthesis algorithm using a Style-Adaptive Layer Norm approach. This maximizes the preservation of the speaker's timbre. Furthermore, a CTC-based alignment framework replaces the MFA in the speech synthesis model training, improving alignment accuracy.

Claims

1. A low-resource speech synthesis system based on BERT feature and style coding, characterized in that, The system includes: An audio acquisition module (100) is configured to acquire audio data to be synthesized, the audio data including Mel spectrum and text data; A Mel spectrum style encoding module (200) is configured to perform timbre prediction processing based on the Mel spectrum to obtain style features; the timbre prediction processing includes global encoding based on the frame-level Mel spectrum and the corresponding temporal sequence of the Mel spectrum; wherein, the Mel spectrum style encoding module (200) includes multiple attention layers arranged in a stacked manner, each attention layer consisting of several activated first attention heads and several second attention heads to be activated, which are set in parallel; the Mel spectrum style encoding module is further configured to divide multiple subsequences based on the Mel spectrum, dynamically determine one or more attention heads corresponding to each subsequence, calculate attention weights for the subsequences based on the one or more attention heads in the attention layer, and complete global encoding based on the attention weights; and adjust the activation state of the second attention heads to be activated in the next level attention layer based on the attention weights output by the previous level attention layer. The BERT text encoding module (300) is configured to perform word and sentence prediction processing on the text data to obtain text features; the word and sentence prediction processing includes sentence encoding prediction, word encoding prediction, concatenation processing based on the results of sentence encoding prediction and word encoding prediction, and text training based on the concatenation processing results; Alignment frame module (400), the alignment frame module (400) is configured to perform pitch alignment prediction processing based on the Mel spectrum and the text data to obtain pitch energy features; A standardization fusion module (500) is configured to perform standardization fusion processing on the style features, the text features and the pitch energy features to obtain target synthesized audio data; The Mel-spectrum style coding module (200) includes: A fully connected spectral unit (210) is configured to convert the Mel spectrum into a sequence of hidden states at the frame level through a built-in fully connected layer. A global feature modeling unit (220) is configured to capture temporal information in the hidden state sequence using a gated convolutional neural network. The temporal information includes the temporal sequence corresponding to each frame number of the hidden state sequence. The gated convolutional neural network includes multiple gated convolutional layers, which are processed by residual connections. The attention mechanism and residual connection unit (230) are composed of multiple attention layers arranged in a stacked manner, with residual connections between the multiple attention layers, and each attention layer is composed of multiple attention heads set in parallel; the attention mechanism and residual connection unit (230) are configured to globally encode the hidden state sequence according to the temporal information to obtain the style features; In the attention mechanism and residual connection unit (230), each attention layer includes one or more preset first attention heads and one or more second attention heads; wherein, the first attention heads are all in an active state, and the second attention heads are configured to adjust at least some of the second attention heads in the current level attention layer to an active state according to the attention weight output by the attention layer of the previous level. Attention weights are calculated for the subsequence based on the first attention head and the second attention head that is in an active state.

2. The low-resource speech synthesis system based on BERT feature and style coding according to claim 1, characterized in that, The attention mechanism and residual connection unit (230) are also configured to: For each frame number of the hidden state sequence, the attention mechanism is calculated multiple times to obtain multiple attention mechanism calculation results corresponding to the same frame number; The average of all attention mechanism calculations for the same frame number is calculated to obtain the average vector. The style features are obtained by summing the residuals of all the average vectors for each frame.

3. The low-resource speech synthesis system based on BERT feature and style coding according to claim 1, characterized in that, The BERT text encoding module (300) is also configured to: Based on the text data, word encoding prediction and sentence encoding prediction are performed respectively to obtain word semantic features and sentence semantic features respectively; The text features are obtained by concatenating the semantic features of the words and the semantic features of the sentences.

4. A low-resource speech synthesis system based on BERT feature and style coding according to claim 3, characterized in that, The BERT text encoding module (300) includes: A BERT text unit (310) is configured to perform sentence encoding prediction on the text data to obtain the sentence semantic features; the sentence encoding prediction is performed at the text sentence level. An encoder text unit (320) is configured to perform word encoding prediction on the text data to obtain the word semantic features; the word encoding prediction is performed at the text word level. A text splicing unit (330) is configured to splice the sentence semantic features and the word semantic features to obtain a spliced ​​text unit, and to perform text training based on the spliced ​​text unit to obtain the text features.

5. A low-resource speech synthesis system based on BERT feature and style coding according to claim 4, characterized in that, The BERT text unit (310) includes several transformer codes, several attention mechanism heads, and several hidden layer units; The encoder text unit (320) includes a high-level feature extraction unit and a fully connected layer.

6. A low-resource speech synthesis system based on BERT feature and style coding according to claim 1, characterized in that, The alignment frame module (400) is also configured to: The Mel spectrum and the text data are convolutionally encoded to obtain the convolutional Mel spectrum and the convolutional text data. Unsupervised alignment calculations are performed on the convolutional Mel spectrum and the convolutional text data to obtain pitch features and energy features; The pitch energy feature includes the pitch feature and the energy feature.

7. A low-resource speech synthesis system based on BERT feature and style coding according to claim 6, characterized in that, The alignment frame module (400) includes: A convolutional coding unit (410) is configured to perform convolutional coding on the Mel spectrum and the text data respectively to obtain convolutional Mel spectrum and convolutional text data; An unsupervised alignment unit (420) is configured to perform unsupervised alignment calculations on the convolutional Mel spectrum and the convolutional text data according to the pairwise L2 distance to obtain the pitch energy features; and to sum the alignment data obtained in the unsupervised alignment calculation process using a forward summation algorithm.

8. A low-resource speech synthesis system based on BERT feature and style coding according to claim 1, characterized in that, The standardized fusion module (500) also includes: A style text standardization fusion unit (510) is configured to: The style features are then normalized to obtain normalized data; The normalized data is scaled and shifted to obtain scaled and shifted data. The scaling-shifted data is processed by a fully connected layer to obtain the activation function and the biased data. The activation function, the paranoia data, the text data, and the pitch energy features are concatenated to obtain synthetic audio data. A decoder unit (520) is configured to decode the synthesized audio data to obtain the target synthesized audio data.

Images