Supervised learning z-TTS generation and quality evaluation system for Tibetan language data enhancement

Abstract

The invention discloses a supervised learning zs-TTS generation and quality evaluation system for Tibetan language data enhancement, and belongs to the technical field of voice intelligent synthesis. The system comprises a preprocessing module, a zero sample text-to-speech synthesis module and a synthesized speech quality evaluation module. The preprocessing module processes the collected Tibetan speech data to obtain a sample; the zero sample text-to-speech synthesis module generates Tibetan speech by using a z-TTS model of supervised learning, introduces a reference encoder on the basis of the TTS model, and optimizes the composition of a loss function of a generative adversarial network; the synthetic speech quality evaluation module scores the synthetic audio in combination with a plurality of indexes, and introduces a dynamic threshold mechanism to balance the quality and number of samples. According to the system, the ability of quickly learning the style of a speaker and generating a high-quality Tibetan speech sample under the condition of extremely short-time input samples is realized, and an effective universal data enhancement scheme is provided for a speech processing task of a low-resource language.

Description

Technical Field

[0001] The invention belongs to the technical field of intelligent speech synthesis, and relates to Tibetan speech data enhancement and speech generation technology, and specifically to a system for generating Tibetan synthesized speech by using a zero-shot text-to-speech (zs-TTS) method based on supervised learning and for enhancing Tibetan task data after screening. Background Art

[0002] In recent years, with the vigorous development of deep learning technology, data enhancement methods have gradually become an important means to improve model performance in the field of speech signal processing. Due to the diversity and complexity of speech signal data in different application scenarios, data enhancement is usually required to improve the generalization and robustness of the model. Common speech data enhancement methods include time domain operations, frequency domain transformations, and noise mixing. These technologies can effectively expand the diversity of training data sets, thereby improving the performance of models in tasks such as speech recognition and speaker recognition. However, the diversity and complex structure of speech signals also bring challenges to data enhancement. How to maximize the use of enhancement methods to enrich data while ensuring semantic consistency is an important research direction for achieving high-precision speech models.

[0003] In the field of speech data enhancement, the combination of traditional methods and emerging technologies has laid the foundation for the improvement of speech processing tasks. Traditional speech data enhancement methods mainly include time domain operations (such as time offset and time stretching), frequency domain transformation (such as frequency masking and frequency offset), noise injection, pitch and speed transformation, etc. These technologies change the time and frequency characteristics of speech signals to simulate different speakers and environments, thereby enhancing the generalization ability of the model. In recent years, with the development of deep learning, the feasibility and research of speech synthesis as a data enhancement method have gradually been recognized. By generating speech samples with different speaking styles and acoustic features, speech synthesis technology can effectively expand the diversity of training data. In addition, by using contrastive learning, the robustness of the model can be further improved by maximizing the similarity of the same speech signal under different enhancement conditions.

[0004] Traditional data enhancement methods usually rely on predefined transformation rules, such as time domain operations and frequency domain transformations. Although these techniques can increase the diversity of data, they are limited in generating speech samples with complex styles and pronunciation characteristics. At the same time, these methods have limited performance in simulating complex environments (such as noisy backgrounds and echo effects), and it is difficult to accurately capture changes in real environments. Even if the sample size is increased through enhancement technology, the generated speech samples may still be unsatisfactory in sound quality and naturalness, thus affecting the training effect of the model. Especially when dealing with low-resource languages ​​such as Tibetan, traditional methods often find it difficult to generate effective speech samples and cannot fully cover language features.

[0005] In recent years, data augmentation methods based on deep learning have made significant progress in the field of speech processing. These methods can improve the diversity and quality of speech data to a certain extent by learning complex audio features and generating models. However, there are still some challenges in the application of low-resource languages. These advanced methods usually rely on large-scale annotated data, while the research of existing technologies mainly focuses on resource-rich languages, resulting in relatively insufficient optimization and research on low-resource languages. In particular, in the application of the zs-TTS method, although this method can generate diverse speech samples in extremely short time samples, its effect in low-resource languages ​​has not been fully verified. This shows that although the zs-TTS method has potential, its practical application on low-resource languages ​​still needs further exploration. Summary of the invention

[0006] In view of the problem that the existing technology is difficult to generate effective speech samples and cannot fully cover language features when generating speech for low-resource languages ​​such as Tibetan, the present invention provides a supervised learning zero-sample text-to-speech generation and quality assessment system for Tibetan data enhancement, which applies the zs-TTS method to Tibetan to generate enhanced samples of different styles.

[0007] The invention provides a supervised learning zero-sample text-to-speech generation and quality assessment system for Tibetan language data enhancement, comprising: a preprocessing module, a zero-sample text-to-speech synthesis module and a synthesized speech quality assessment module.

[0008] The preprocessing module performs preprocessing operations on the collected Tibetan speech data, including Tibetan text preprocessing and Tibetan speech preprocessing, to obtain Tibetan speech data samples, each sample containing an audio waveform x, a Mel spectrum m corresponding to the audio waveform x, a sentence text t, and a speaker number s.

[0009] The zero-shot text-to-speech synthesis module generates Tibetan speech using the zs-TTS model of supervised learning. The zs-TTS model includes a generative adversarial network, a random duration predictor, a reference encoder, a text encoder, a posterior encoder, and a residual coupling module. The Tibetan speech sample (x, m, t, s) is input into the zero-shot text-to-speech synthesis module; the reference encoder captures the speaker's style features from the given reference audio and outputs a feature embedding vector; the text encoder encodes the input sentence text t and extracts the text feature representation; the posterior encoder converts the input Mel spectrum m into a latent vector and outputs it to the residual coupling module, which uses the latent variables output by the posterior encoder and the feature embedding vector output by the reference encoder to generate a fusion representation containing the speaker's style. The random duration predictor receives the text features output by the text encoder and the feature embedding vector output by the reference encoder to predict the duration of each phoneme, ensuring that the generated audio duration is consistent with the text. The generative adversarial network includes a decoder and a multi-cycle discriminator. The decoder performs speech synthesis based on the fused representation output by the residual coupling module and the phoneme duration predicted by the random duration predictor to generate a target audio waveform, which is then input together with the real audio waveform into the multi-period discriminator to distinguish between true and false audio.

[0010] A dynamic threshold mechanism is set in the synthetic speech quality assessment module to calculate the weighted scores of subjective speech quality evaluation pseq, cosine similarity cs, mean square error mse and word error rate wer for the target audio waveform and compare them with the current threshold θ final Compare, if it is greater than θ final The target audio waveform is considered to be of qualified quality, otherwise the target audio waveform is considered to be unqualified and discarded. The initial threshold θ is set in the dynamic threshold mechanism base and adjustment coefficient α, calculate the current threshold θ final as follows:

[0011] θ final =max(θ,θ min );

[0012] θ=θ base +α·(Q avg -Q base );

[0013] Among them, θ min is the lower threshold, Q avg is the average quality score of the current sample, Q base is a preset quality benchmark score.

[0014] In the zero-sample text-to-speech synthesis module, a training sample set is used to train a generative adversarial network in advance, and the training samples are input into the zero-sample text-to-speech synthesis module. The decoder generates a target audio waveform, and the multi-cycle discriminator performs multi-scale discrimination on the generated target audio waveform at different cycles. During training, the loss functions of the decoder and the multi-cycle discriminator are set as follows to update the network parameters.

[0015] The decoder loss function L D Calculated as: Where B is the number of samples, x i is the audio waveform of the ith sample, z i is the fusion feature output by the residual coupling module for the i-th sample, is the phoneme duration vector predicted by the random duration predictor for the i-th sample, G represents the decoder, and D represents the multi-cycle discriminator.

[0016] The loss function L of the multi-cycle discriminator gen_all Including the decoder loss L gen , Mel spectrum loss L mel , feature matching loss L fm , duration loss L dur and KL divergence loss L kl , L gen_all Calculated as:

[0017] L gen_all =L gen +c mel ·L mel +L fm +L dur +c kl ·L kl ;

[0018] Among them, c mel and c kl are the weight coefficients of Mel spectrum loss and KL divergence loss respectively.

[0019] The multi-cycle discriminator is implemented based on the improved HiFi-GAN network, including a one-dimensional convolution discriminator DiscriminatorS and a multi-cycle two-dimensional convolution discriminator DiscriminatorP. The depth of feature extraction is improved by increasing the number of convolution layers, including: deepening DiscriminatorS from 6 one-dimensional convolution layers to 8 one-dimensional convolution layers, gradually increasing the input channels from 1 to 2048, reducing the convolution kernel size from 15 to 5, adjusting the stride and padding accordingly, and finally outputting 1 channel; deepening DiscriminatorP from 5 two-dimensional convolution layers to 7 two-dimensional convolution layers, increasing the input channels from 1 to 2048 layer by layer, keeping the convolution kernel and padding consistent, and finally outputting 1 channel.

[0020] The duration loss is calculated by presetting different weights for different phonemes, setting higher weights for important phonemes used in daily Tibetan, using the weights as the weights of the phoneme duration differences, and obtaining the duration loss by calculating the weighted mean square error between the actual phoneme duration of the sample and the predicted phoneme duration.

[0021] Compared with the prior art, the advantages and positive effects of the present invention are:

[0022] (1) The system of the present invention introduces Zero-Shot Text-to-Speech (zs-TTS) technology, optimizes the loss function design, and improves the quality of synthesized speech. The model can generate diverse Tibetan speech samples under the condition of extremely short input samples combined with the reference encoder, filling the research gap in low-resource language data enhancement, especially in low-resource languages ​​such as Tibetan, and improving the diversity and effectiveness of data.

[0023] (2) The system of the present invention introduces a synthetic speech quality assessment module driven by ensemble learning, which automatically identifies and retains high-quality synthetic samples to ensure the effect of data enhancement; the overall quality of the samples is judged by weighted scores, and high-quality samples can be accurately screened out; a dynamic threshold adjustment mechanism is introduced to ensure that the quality and quantity of the generated samples are balanced within a certain range, optimize the overall performance of the model, and improve the accuracy and reliability of the evaluation of synthetic speech samples.

[0024] (3) The system of the present invention can be used as a general solution for data enhancement of low-resource languages. It is not only applicable to Tibetan, but can also be extended to other low-resource languages, such as Uyghur and Mongolian. Experimental verification shows that the system of the present invention has significant advantages in improving the quality and diversity of speech data of these languages, which helps to improve the training effect of related tasks, including speech recognition and speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a main framework implementation diagram of the zs-TTS generation and quality assessment system of an embodiment of the present invention;

[0026] Figure 2 This is a graph comparing the audio quality of the zs-TTS model of the present invention and other traditional TTS models;

[0027] Figure 3 This is a comparison chart of the effects of the method of the present invention and other data enhancement methods on word error rate. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] The system of the present invention implements the zs-TTS technology of supervised learning for low-resource languages. By introducing a reference encoder based on the TTS model, optimizing the loss function composition of the generative adversarial network, and using a multi-speaker Tibetan data set for training, the system can quickly learn the speaker style and generate high-quality Tibetan speech samples in the case of extremely short-term input samples. At the same time, the present invention also designs a synthetic speech quality assessment module based on ensemble learning, combines multiple indicators such as subjective speech quality evaluation (PESQ), cosine similarity (Cosine Similarity), mean square error (MSE) and word error rate (WER) for comprehensive evaluation, and introduces a dynamic threshold mechanism to balance sample quality and quantity. The system of the present invention not only improves the effect of Tibetan data enhancement, but also provides an effective general data enhancement solution for speech processing tasks of low-resource languages.

[0030] The embodiment of the present invention takes Tibetan, a low-resource language, as an example to illustrate the implementation of a supervised learning zs-TTS generation and quality assessment system for Tibetan data enhancement. The application environment used is: Ubuntu system, python3.8, pytorch deep learning architecture. The supervised learning zs-TTS generation and quality assessment system for Tibetan data enhancement of the present invention includes a preprocessing module, a zero-sample text-to-speech synthesis module, a synthesized speech quality assessment module and a speech recognition module. The improvement of the present invention mainly lies in the zero-sample text-to-speech synthesis module and the synthesized speech quality assessment module, such as Figure 1 shown.

[0031] The preprocessing module is mainly responsible for the preliminary processing and preparation of Tibetan speech data. The preprocessing module first converts the audio format of the Tibetan speech data of multiple speakers into wav format, then resamples, performs bandpass filtering, removes electrooculographic interference, removes too long or too short audio and other denoising processes to ensure the quality of the speech data, and extracts Mel spectrum features. Then the Tibetan text is preprocessed. The preprocessing operations mainly include converting Tibetan into Wiley transcription format and unifying text representation; cleaning text to clean up non-text content such as redundant characters and punctuation; then performing spelling correction to correct errors and irregular usage to improve the standardization of the text; and standardizing professional terms by building a dictionary to ensure accurate expression of terms. Finally, the data is divided into training set, validation set and test set for use in subsequent model training.

[0032] In the embodiment of the present invention, the preprocessing module preprocesses the collected Tibetan speech data and divides them into training set, verification set and test set. The number of batches is divided according to the set number of samples, and each batch of data includes an audio waveform X, a corresponding Mel spectrum M, a corresponding text label T and a corresponding speaker number S, which are expressed as follows:

[0033] (X, M, T, S) = {(x i , m i , t i ,s i )|i∈{1,2,…,B}}

[0034] Where B is the number of samples in the batch, x i is the audio waveform of the i-th sample, each x i Corresponding to a Mel spectrum m i , a text label t i and a speaker number s i ,s i The value range is {1, 2, ..., K}, where K is the total number of speakers. The text tag records the converted and standardized text sentences. Figure 1 As shown, spectrum extraction refers to extracting Mel spectrum from audio waveform.

[0035] The zero-shot text-to-speech synthesis module is the core of the entire system, and uses the zs-TTS model of supervised learning to generate Tibetan speech. By introducing a reference encoder, the zs-TTS model can quickly learn the speaker's style characteristics under extremely short input sample conditions and generate diversified Tibetan speech samples that match it. The zero-shot text-to-speech synthesis module uses a generative adversarial network (GAN) to train the generator, and further optimizes the generator's loss function, combined with the weighting of multiple loss terms, to improve the quality and diversity of the synthesized speech. Compared with traditional signal processing-based methods or standard TTS methods, the zero-shot text-to-speech synthesis module of the present invention demonstrates significant advantages in generating diverse, high-quality speech samples, and provides a new solution for data enhancement of low-resource languages.

[0036] like Figure 1 As shown, the implementation structure of the zero-sample text-to-speech synthesis module includes a generative adversarial network (GAN), a random duration predictor, a reference encoder, a text encoder, a posterior encoder, and a residual coupling module. The generative adversarial network includes a generator G and a discriminator D, and the present invention improves it by setting a random duration predictor and a reference encoder. Figure 1 The decoder is the generator G, and the multi-cycle discriminator is the discriminator D.

[0037] In terms of model design, the present invention introduces a stochastic duration predictor and a reference encoder to achieve more accurate phoneme duration prediction and spectrogram encoding. The integration of these modules not only improves the temporal resolution of the model at the phoneme level, but also enhances the adaptability to different speaker characteristics and background noise, thereby significantly improving the naturalness and fluency of speech synthesis. The stochastic duration predictor predicts the duration of each phoneme in the text based on the input text features and speech style features, so that the generated speech samples are more diverse and natural in duration, so as to help the decoder generate more realistic speech. The reference encoder captures the speaker's style features from the spectral features of the given reference audio and generates an implicit representation for guiding the synthesis process. The introduction of this module effectively improves the model's ability to capture the target speaker's style, making the synthesized speech not only more accurate in spectral features, but also able to retain the unique speech features of the target speaker. In addition, the reference encoder also enhances the generalization ability of the model in a low-resource environment, and the model can generate high-quality speech output even when the training data is limited.

[0038] like Figure 1 As shown in the figure, the reference encoder captures the features of the input speech from the Mel spectrum of the input audio and generates an embedding vector of a fixed size. The core structure of the reference encoder includes convolutional layers and GRU (gated recurrent unit structure) layers. The input speech spectrogram is processed by 6 layers of two-dimensional convolution. Each layer extracts features at different levels, gradually compresses the time dimension and frequency resolution, and increases the number of channels to extract richer feature information. Subsequently, these convolution-processed features are input into the GRU layer to further process the timing information and finally output an embedding vector containing the speech style. This embedding vector is used to generate speech samples consistent with the input speech style and serves as conditional information to guide the training of subsequent text encoders, residual coupling modules, and GAN structures.

[0039] like Figure 1 As shown, for each sample (x i ,m i ,t i ,s i ), the text encoder is used to input the text t iExtract text features to capture the content information required for speech generation. The posterior encoder is responsible for converting the Mel spectrum of the audio waveform into a latent vector as the posterior distribution representation of the speech features. Through this module, the zs-TTS model can learn the precise details of the speech. The residual coupling module is a flow model, and each coupling layer consists of a residual coupling layer and a flip layer. The residual coupling module combines the speech style features output by the reference encoder and the latent vector output by the posterior encoder to generate a fusion feature representation that conforms to the input speech and reference audio style, thereby effectively enhancing the quality and naturalness of the generated speech.

[0040] The decoder performs speech synthesis based on the fused feature representation output by the residual coupling module and the duration of each phoneme predicted by the random duration predictor, generates the target audio waveform, and inputs the target audio waveform and the real audio waveform into the multi-period discriminator to distinguish between real and fake audio. In order to achieve the rhythm and naturalness of speech, the multi-period discriminator uses different sampling periods to segment speech signals of different lengths, and performs multi-scale discrimination on the generated speech at different periods to ensure that the speech generated by the decoder is realistic and natural in all frequency ranges. The decoder and the multi-period discriminator compete with each other, that is, the decoder generates sufficiently realistic speech to deceive the multi-period discriminator, while the multi-period discriminator improves the ability to distinguish fake audio, making the generated speech closer to real audio.

[0041] In the embodiment of the present invention, for the input sample (x i , m i , t i ,s i ) and a given reference audio, each module is calculated as follows:

[0042] Reference encoder output z r =E ref (y ref_mel ), E ref is the reference encoder, y ref_mel is the Mel-spectrogram of the reference audio, z r is the extracted feature vector containing the speaker's voice style.

[0043] The output of the text encoder is text =E text (z r , t i ), E text is a text encoder, e text Is a text encoding representation.

[0044] The output z of the posterior encoder posterior =E posterior (m i ), E posterior is the posterior encoder, z posterioris the latent vector encoded by the posterior encoder.

[0045] The output z of the residual coupling module i =Coupling(z r , z posterior ), Coupling is the residual coupling module, z i It is the fusion feature representation.

[0046] Output of the random duration predictor P duration is a random duration predictor.

[0047] Decoder Output G is the decoder, yi is the generated target audio waveform, is the predicted phoneme duration vector.

[0048] In order to improve the generation quality and diversity of speech synthesis models, the present invention designs a training process based on generative adversarial networks, and realizes efficient training through joint optimization of multiple loss functions. First, batch data (X, M, T, S) is obtained from the data loader, and the obtained batch data is processed on the GPU to speed up the calculation. In the forward propagation stage, the generator G generates a target audio waveform based on the input fusion feature representation and phoneme duration, and then inputs it together with the real audio waveform into the discriminator D to distinguish between true and false audio. The real audio waveform is the audio waveform in the input sample. Suppose the sample (x i , m i , t i ,s i ) Get the fusion feature z i and the predicted phoneme duration vector The loss of the discriminator D is calculated by the identification loss L D Get, and back propagate the optimization parameters, identify the loss L D The calculation is as follows:

[0049]

[0050] D(x i ), Denotes the discriminator D for x i , the generated target audio waveform y i Identify true and false audio.

[0051] The generator G passes the generator loss L gen , Mel spectrum loss L mel , feature matching loss L fm , duration loss L dur and KL divergence loss L klThe weighted sum of these five loss terms is used to evaluate the generated effect, that is, the custom loss function L gen_all The generator is optimized and calculated as follows:

[0052] L gen_all =L gen +c mel ·L mel +L fm +L dur +c kl ·L kl

[0053] Among them, c mel and c kl is the weight coefficient of Mel spectrum loss and KL divergence loss.

[0054] The generator hopes to deceive the discriminator so that the probability that the discriminator identifies the generated audio as "real" is as close to 1 as possible, so the generator loss L gen It is measured by calculating the mean square error between the generator output and the target 1. Mel spectrum loss L mel It is obtained by calculating the L1 distance between the Mel spectrum of the generated audio and the real audio, so as to optimize the spectral features of the generated audio and make it as close to the real audio as possible. Feature matching loss L fm Calculate the absolute difference between the feature maps of the real audio extracted from the discriminator and the generated audio, use the absolute value function to calculate the difference between each element, and then average the average absolute difference of each layer. The average absolute difference of all layers is summed to get the total feature matching loss. Duration loss L dur It is obtained by calculating the mean squared difference between the actual phoneme duration and the predicted phoneme duration. The KL loss calculates the difference between the original feature representation and the feature representation output after the posterior encoder, using its mean and logarithmic variance, and obtains the final value through weighted summation and normalization.

[0055] The system of the present invention adopts a multi-cycle discriminator based on the HiFi-GAN network improvement, including a one-dimensional convolution discriminator (DiscriminatorS) and a multi-cycle two-dimensional convolution discriminator (DiscriminatorP), and extracts features through different convolution levels for integration. This multi-scale feature fusion capability can significantly enhance the accuracy of the model in identifying subtle differences in synthetic audio. The speech characteristics of Tibetan are different from those of other languages, including the pronunciation characteristics of phonemes, tone changes, and syllable structures. For this reason, the system of the present invention increases the depth of feature extraction and enriches the diversity of feature representation by increasing the number of convolution layers. The deepened DiscriminatorS is deepened from 6 one-dimensional convolution layers to 8 one-dimensional convolution layers, the input channel is gradually increased from 1 to 2048, the convolution kernel size is reduced from 15 to 5, the stride and padding are adjusted accordingly, and the final output channel is 1. DiscriminatorP is deepened from 5 two-dimensional convolution layers to 7 two-dimensional convolution layers, the input channel increases from 1 to 2048 layer by layer, the convolution kernel and padding remain consistent, and finally outputs 1 channel. This structural design enhances the model's feature extraction capabilities. In the calculation of feature matching loss, these deep audio features are used to help the model more accurately simulate the feature distribution of real audio signals. By comparing the features of generated audio with those of real audio, the model can more effectively identify subtle differences in the synthesis process, ultimately improving the naturalness and fluency of synthesized speech.

[0056] Duration loss is designed to ensure that the model can accurately predict the duration of Tibetan phonemes. In this process, the present invention specifically weights important phonemes in daily Tibetan use, such as vowels and consonants unique to Tibetan. The core of this weighting strategy is to give higher weights to these key phonemes, thereby making the model's predictions on them more accurate. Specifically, certain vowels and consonants in the Tibetan phoneme system have higher recognition and importance in speech communication, especially when expressing word meaning or grammatical functions. For example, vowels and consonant and and The pronunciation difference will affect the meaning of words, etc. Therefore, when calculating the duration loss, different weights are assigned to different phonemes in advance, and the weights corresponding to these important phonemes are set higher. When calculating the duration loss, the set weights are used to weight the phoneme duration difference, and the duration loss is obtained by calculating the weighted mean square error between the real phoneme duration and the predicted phoneme duration of the sample, so that the model can pay more attention to the duration prediction of important phonemes during the learning process. Through this weighted processing, the model is more inclined to optimize the prediction accuracy of these important phonemes during training, thereby effectively improving the naturalness and fluency of the synthesized speech. This method not only enhances the model's sensitivity to Tibetan phonemes, but also provides a more solid foundation for subsequent speech synthesis, ensuring that the synthesis results can better reflect the pronunciation characteristics and speech habits of Tibetan.

[0057] like Figure 2 As shown in the figure, the quality of synthesized audio of the zs-TTS model implemented in the embodiment of the present invention is compared with a variety of traditional TTS models, including Tacotron2, Fastspeech, Transformer TTS and VITS models, and the mean opinion score (MOS) and relative mean opinion score (CMOS) indicators are calculated respectively. GT represents the MOS score of the real audio. Figure 2 It can be seen that the audio quality synthesized by the zs-TTS model of the present invention is very close to the audio quality of the traditional single-speaker TTS model, which verifies the effectiveness of the model.

[0058] In order to improve the quality of the synthesized speech used, the present invention designs a synthesized speech quality assessment module based on ensemble learning. The module combines multiple indicators such as subjective speech quality evaluation pseq, cosine similarity cs, mean square error mse and word error rate wer to comprehensively assess the quality of the synthesized sample through weighted scores to ensure the effectiveness of data enhancement. The synthesized speech quality assessment module receives the synthesized speech sample and the real speech sample, and converts them into the corresponding Mel-spectrogram M and pseq is used for speech quality assessment. It uses the calculation function encapsulated by the PESQ toolkit and inputs the M and Generate a score. Cosine similarity cs measures spectral similarity. Mean square error mse evaluates detail restoration ability. Word error rate wer detects speech recognition accuracy, and the relevant definitions are as follows:

[0059]

[0060] in C is the number of Mel frequency channels, T is the time step, and R represents a set of real numbers. c,t , They are M, The value of the Mel frequency at channel c at time step t.

[0061] The synthetic speech quality assessment module compares the effects of multiple ensemble learning methods and finally uses a weighted method to calculate the comprehensive score Q to improve the accuracy and robustness of the assessment. The weighted operation is defined as the score_fusion function, which is calculated as follows:

[0062]

[0063] where λ1, λ2, λ3 and λ4 represent the relative weight of each score.

[0064] The comprehensive score is used to screen samples that meet the quality standards and integrate them into the Tibetan multi-speaker dataset to support model training and optimization. In this way, the synthetic speech quality assessment module can provide an accurate judgment of the overall quality of the generated samples, ensuring the generation of high-quality synthetic speech and improving the effect of data enhancement.

[0065] In order to ensure that the generated speech samples meet both the quality standard and the quantity requirement, the method of the present invention introduces a dynamic threshold mechanism in the speech synthesis task, and automatically adjusts the threshold θ according to the quality and quantity requirements of the generated samples. The relevant definitions are as follows:

[0066] θ=θ base +α·(Q avg -Q base )

[0067] where θ base is the initial threshold, α is the adjustment coefficient, Q avg is the average quality score of the current sample, Q base is a preset quality benchmark score.

[0068] At the same time, set the lower threshold θ min , the final threshold θ final Calculated by the following formula:

[0069] θ final =max(θ,θ min )

[0070] In practical applications, by introducing dynamic threshold adjustment and lower limit setting mechanism, the quality and quantity of samples can be effectively balanced to ensure that the generated samples meet the quality requirements and have sufficient quantity for subsequent analysis and model training. This method helps to improve the effect of data enhancement and optimize the overall quality of synthesized speech. When the quality score Q corresponding to the sample is greater than θ final When Q is less than or equal to θ, the quality of the synthesized audio is considered qualified and meets the standards for subsequent use. final, the audio is considered to be of substandard quality and will be discarded.

[0071] The embodiment of the present invention applies the quality-assessed synthetic speech samples to the training of the Tibetan speech recognition (ASR) model, and verifies its effect on improving the model performance. First, the screened high-quality synthetic speech samples are mixed with the original samples in different proportions and used to train the Tibetan ASR model to explore the influence of the sample ratio on the model training effect. In data preparation, the synthetic audio is only integrated into the training set, while the test set and the validation set still retain the original audio data to ensure the fairness of the evaluation and the generalization ability of the model. Optimization strategies are applied during the model training process, including data enhancement and model fine-tuning. In order to comprehensively evaluate the effect of the new method, the embodiment of the present invention compares the synthetic speech data enhancement method proposed by the method of the present invention with the traditional signal processing-based and traditional TTS (Text-to-Speech)-based data enhancement methods.

[0072] The experimental design includes five repeated tests, each using a different random seed. The data set is split into 60%, 20%, and 20%, where the synthetic audio will only be integrated into the training set. The original training set is expanded in scale, with 2x, 3x, and 4x representing the ratio of synthetic speech to real audio of 2:1, 3:1, and 4:1. In addition, this experiment also explores the effect of the speech recognition model trained using different data augmentation methods after migration to the new data distribution, corresponding to 4x Post-migration. The effect of the method of the present invention on word error rate is compared with other data augmentation methods. Figure 3 As shown. In the actual test, the zs-TTS model of the present invention including the synthetic speech quality assessment module is marked as zs-TTS-Ensemble, and compared with the method of data enhancement using the zs-TTS model of the present invention alone without the synthetic speech quality assessment module. At the same time, the performance of the model of the present invention is compared with 8 common data enhancement methods, including traditional data enhancement methods such as Speed ​​perturbation, Specaugment, Noise Augmentation, VolumeAugmentation, and TTS-based data enhancement methods such as Tacotron2+WaveGlow, Fastspeech+WaveGlow, Transformer TTS (Mel+WaveGlow), and VITS. Raw is the case where no data enhancement method is used and the original data set is expanded by copying.

[0073] Depend on Figure 3It can be seen that the model proposed by the system of the present invention has achieved the best performance in comparison with 8 common data enhancement methods, and the speech recognition model trained by the TTS-based data enhancement method has better performance in migration to new data distribution than the traditional data enhancement method, and the accuracy rate drops less after migration. The model proposed by the system of the present invention has achieved the smallest drop of -11.8%. It can be seen that the system of the present invention has significant advantages in generating diversified and high-quality speech samples, and provides a new solution for data enhancement of low-resource languages.

[0074] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A supervised learning zs-TTS generation and quality assessment system for Tibetan language data enhancement, characterized in that: It includes a preprocessing module, a zero-sample text-to-speech synthesis module, and a synthesized speech quality assessment module; zs-TTS stands for zero-sample text-to-speech; The preprocessing module performs preprocessing operations on the collected Tibetan speech and Tibetan text data to obtain Tibetan speech data samples, each sample including an audio waveform x, a Mel spectrum m corresponding to the audio waveform x, a sentence text t, and a speaker number s; The zero-sample text-to-speech synthesis module generates Tibetan speech using a supervised learning zs-TTS model; the zs-TTS model includes a generative adversarial network, a random duration predictor, a reference encoder, a text encoder, a posterior encoder, and a residual coupling module; the Tibetan speech sample (x, m, t, s) is input into the zero-sample text-to-speech synthesis module; the reference encoder captures the speaker's speech style features from a given reference audio and outputs a feature embedding vector; the text encoder performs feature encoding on the sentence text t to obtain text features; the posterior encoder converts the Mel spectrum m into a latent vector; the residual coupling module generates a fusion feature containing the speaker's speech style using the latent variable and the feature embedding vector; the random duration predictor predicts the duration of each phoneme based on the text features and the feature embedding vector; the generative adversarial network includes a decoder and a multi-cycle discriminator, the decoder performs speech synthesis based on the fusion features and the predicted duration of each phoneme to generate a target audio waveform, and the target audio waveform is input into the multi-cycle discriminator together with the real audio waveform to distinguish between true and false audio; A dynamic threshold mechanism is set in the synthetic speech quality assessment module to calculate the weighted scores of subjective speech quality evaluation pseq, cosine similarity cs, mean square error mse and word error rate wer for the target audio waveform and compare them with the current threshold θ final Compare, if it is greater than θ final If the target audio waveform is good, the target audio waveform is considered to be of good quality; otherwise, the target audio waveform is considered to be unqualified and discarded; The dynamic threshold mechanism sets an initial threshold θ base and adjustment coefficient α, calculate the current threshold θ final as follows: i final =max(θ,θ min ); θ=θ base +α·(Q avg -Q base ); Among them, θ min is the lower threshold, Q avg is the average quality score of the current sample, Q base is a preset quality benchmark score.

2. The system according to claim 1, characterized in that The preprocessing module performs preprocessing operations on Tibetan speech, including resampling, bandpass filtering, removing electrooculographic interference, and removing audio that does not meet the length requirements. The preprocessing operations on Tibetan text include Willy transcription, text cleaning, spelling correction, and standardized professional terminology. The preprocessed Tibetan speech and text data sample data are divided into a training set, a verification set, and a test set.

3. The system according to claim 1, characterized in that The reference encoder includes a convolution layer and a gated recurrent unit structure GRU layer. The Mel spectrum of the input audio is first processed by 6 layers of two-dimensional convolution to extract features at different levels. The extracted features are then input into the GRU layer to extract audio timing information, and finally a speech style feature is output.

4. The system according to claim 1, characterized in that In the zero-sample text-to-speech synthesis module, a training sample set is used to train a generative adversarial network in advance, and the training sample is input into the zero-sample text-to-speech synthesis module, a decoder generates a target audio waveform, and a multi-cycle discriminator performs multi-scale discrimination on the generated target audio waveform at different cycles; during training, the loss functions of the decoder and the multi-cycle discriminator are set as follows to update the network parameters; The decoder loss function L D Calculated as: Where B is the number of samples, x i is the audio waveform of the ith sample, which is the real audio waveform, z i is the fusion feature output by the residual coupling module for the i-th sample, is the phoneme duration vector predicted by the random duration predictor for the i-th sample, G represents the decoder, and D represents the multi-cycle discriminator; The loss function L of the multi-cycle discriminator gen_all Including the decoder loss L gen , Mel spectrum loss L mel , feature matching loss L fm , duration loss L dur and KL divergence loss L kl , L gen_all Calculated as: L gen_all =L gen +c mel ·L mel +L fm +L dur +c kl ·L kl ; Among them, c mel and c kl are the weight coefficients of Mel spectrum loss and KL divergence loss respectively.

5. The system according to claim 1 or 4, characterized in that: The multi-cycle discriminator is implemented based on the improved HiFi-GAN network, including a one-dimensional convolution discriminator DiscriminatorS and a multi-cycle two-dimensional convolution discriminator DiscriminatorP. The depth of feature extraction is improved by increasing the number of convolution layers, including: deepening DiscriminatorS from 6 one-dimensional convolution layers to 8 one-dimensional convolution layers, gradually increasing the input channels from 1 to 2048, reducing the convolution kernel size from 15 to 5, adjusting the stride and padding accordingly, and finally outputting 1 channel; deepening DiscriminatorP from 5 two-dimensional convolution layers to 7 two-dimensional convolution layers, increasing the input channels from 1 to 2048 layer by layer, keeping the convolution kernel and padding consistent, and finally outputting 1 channel.

6. The system according to claim 4, characterized in that The duration loss is calculated by presetting different weights for different phonemes, setting higher weights for important phonemes used in daily Tibetan, using the weights as the weights of the phoneme duration differences, and obtaining the duration loss by calculating the weighted mean square error between the actual phoneme duration of the sample and the predicted phoneme duration.

Images