A method, system, device and medium for Tibetan speech synthesis

Abstract

The application discloses a Tibetan speech synthesis method, system, device and medium, and relates to the technical field of speech synthesis. The method comprises the following steps: acquiring original ASR audio and corresponding Tibetan text; performing end-to-end repair processing on the original ASR audio to obtain repaired audio; performing byte pair encoding on the Tibetan text to obtain a plurality of text features; and labeling the syllable structure corresponding to each text feature to obtain a text condition feature; training an acoustic model based on conditional flow matching through the repaired audio and the text condition feature to obtain a speech synthesis model; and inputting target text and reference audio into the speech synthesis model to obtain Tibetan synthesized speech that conforms to the target text and matches the acoustic features of the reference audio. The application enables the acoustic model based on conditional flow matching to more accurately learn the time length distribution law, prosodic pause mode and collaborative pronunciation rules specific to the Tibetan language, thereby greatly improving the adaptability of Tibetan speech to personalized application scenarios.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a Tibetan speech synthesis method, system, device and medium. Background Technology

[0002] Deep learning has driven the rapid development of text-to-speech (TTS) technology, achieving breakthroughs in the naturalness, intelligibility, and expressiveness of synthesized speech. While near-human-level speech can be generated for high-resource languages ​​such as English and Chinese, low-resource languages ​​like Tibetan, with their unique linguistic features and complex writing systems, still face significant challenges.

[0003] Currently, scholars have proposed a low-resource Tibetan speech synthesis system (Xizang-TTS) based on large-scale model adaptation. Its technical implementation includes: constructing a unified data quality enhancement process; performing audio cleaning, text regularization, and speech-text consistency verification on low-quality, multi-source Tibetan speech data. Addressing the mismatch between the complex structure of the Tibetan writing system and the default segmentation strategy of the pre-trained model, two adaptation strategies were explored: one is to maintain the original segmenter but adjust the input representation; the other is to replace it with a BPE segmenter trained on a Tibetan corpus, allowing the input representation to better align with Tibetan natural language units. Using a pre-trained large-scale model, cross-lingual representation and generation capabilities are learned from large-scale mixed Chinese-English speech data and multi-dialect speech data. Combined with a lightweight fine-tuning strategy, cross-lingual adaptive fine-tuning is performed on high-quality Tibetan speech-text pairing data to obtain synthesized Tibetan speech.

[0004] However, the lightweight fine-tuning strategy of this existing technology solidifies the underlying cross-language representation of the pre-trained model, so that all training data are standardized into a single reading mode after uniform cleaning, resulting in the trained model being able to only reproduce a specific style and making it difficult to adapt to personalized application scenarios. Summary of the Invention

[0005] Therefore, it is necessary to provide a Tibetan speech synthesis method, system, device, and medium to address the aforementioned technical problems.

[0006] This invention provides a Tibetan speech synthesis method, comprising: Obtain the original ASR audio and its corresponding Tibetan text; The original ASR audio is repaired end-to-end using a neural network-based speech restoration model to restore damaged acoustic features and reconstruct high-fidelity time-domain waveforms, resulting in restored audio. The Tibetan text is encoded byte-to-byte using the Xizang-Alpaca word segmenter to compress the original Tibetan character sequence into a sub-word sequence that retains syllable structure information, resulting in multiple text features representing sub-word features. The syllable structure corresponding to each text feature is annotated using the G2P module to provide syllable boundary information and pronunciation category differentiation annotations, resulting in text conditional features. The acoustic model based on conditional flow matching is trained by repairing the conditional features of audio and text to learn the mapping relationship from Gaussian noise distribution to Mel spectrum distribution, thus obtaining a speech synthesis model. The target text and reference audio are input into the speech synthesis model to obtain Tibetan synthesized speech that matches the acoustic features of the target text and the reference audio.

[0007] Optionally, the original ASR audio is subjected to end-to-end restoration processing using a neural network-based speech restoration model to repair damaged acoustic features and reconstruct high-fidelity time-domain waveforms, resulting in restored audio, specifically including: Neural network-based speech restoration models include ResUNet and TFGAN vocoders; By analyzing the deep convolutional feature extraction and skip connection mechanism of the cross-scale receptive field of the ResUNet network, nonlinear modeling of the damaged acoustic features is performed to obtain a Mel spectrogram that is closer to the pure speech. Based on the Mel spectrogram, high-fidelity time-domain speech was reconstructed using an adversarial waveform via a TFGAN vocoder, resulting in the restored audio.

[0008] Optionally, after obtaining the restored audio, the process also includes multi-dimensional filtering of the restored audio, specifically including: Obtain the speech segment and silence segment in each repaired audio, and determine the first average energy of the speech segment and the second average energy of the silence segment respectively; The signal-to-noise ratio is determined based on the first average energy and the second average energy, and the repaired audio corresponding to the signal-to-noise ratio being lower than a preset threshold is removed to obtain multiple retained audio; The root mean square amplitude of each preserved audio track is normalized to the target root mean square amplitude based on the preset overall gain coefficient, thus obtaining the preliminary speech. The speech boundaries of the preliminary speech are identified by a pre-trained speech activity detection model, and the beginning and end silences are trimmed to obtain the final speech. The number of syllables per second in the final language is compared with the preset syllable interval to eliminate the final speech that does not meet the preset syllable interval, thus obtaining the filtered and repaired audio.

[0009] Optionally, before encoding the Tibetan text byte-to-byte using the Xizang-Alpaca word segmenter, the Tibetan text undergoes normalization processing, specifically including: Constructing a text normalization cascade converter based on a weighted finite state converter; The input text is deterministically mapped by a text normalization cascade converter through number conversion rules, date conversion rules, symbol conversion rules, and abbreviation conversion rules to obtain normalized Tibetan text, and pause level labels and boundary labels are output simultaneously.

[0010] Optionally, the Tibetan text is byte-pair encoded using the Xizang-Alpaca word segmenter to compress the original Tibetan character sequence into a subword sequence that preserves syllable structure information, resulting in multiple text features representing subword characteristics, specifically including: The Tibetan text is split according to the original character order to obtain the initial symbol sequence; Get the built-in byte pair merging rule table of the Xizang-Alpaca word segmenter. The merging rule table contains a sequence of merging operations sorted by priority. Adjacent symbol pairs are merged in descending order of merging priority. After each merge, the current symbol sequence is updated and the merging priority of adjacent symbol pairs is recalculated. The merging, updating and counting operations are repeated. Merging stops when there are no adjacent symbol pairs that can be merged in the current symbol sequence, resulting in the BPE subword sequence. Each sub-word unit in the BPE sub-word sequence is mapped to a corresponding lexical number, and the correspondence between each lexical and the original Tibetan characters and their corresponding syllables in the original Tibetan text is preserved; By inputting the lexical numbers and their positional encodings into the text encoder, multiple text features are obtained.

[0011] Optionally, the syllable structure corresponding to each text feature is labeled using the G2P module to provide syllable boundary information and pronunciation category differentiation annotations, thereby obtaining text conditional features, specifically including: For each BPE sub-word unit in the text features, reconstruct the Tibetan character fragment corresponding to that BPE sub-word unit; The Tibetan character fragments are segmented into syllables, and the superposition structure of each syllable is parsed according to the preset Tibetan syllable slot template. The base character, preceding character, superscript character, underscript character, suffix character, subsequent suffix character, vowel symbol and superscript character symbol of the Tibetan character fragment are used to obtain structured labels describing the syllable slots and their combination relationships. Based on the syllable slots and their combination relationships in the structured tags, generate corresponding phoneme tags and G2P structure tags for each character or character position, and preserve character order, syllable boundaries, and pairing relationships between characters and phoneme slots; For Tibetan syllables with the same base word but different superposition structures, different G2P conditions are generated based on their slot combinations and structural tags to distinguish the corresponding pronunciation and tone variants. For words not registered in the database, the preset rules drive the pronunciation backtracking module to deduce the corresponding phoneme sequence based on its character composition, syllable slot structure and Tibetan spelling rules, and use the derivation result as a G2P annotation. The lexical number representation, positional encoding, and position-by-position G2P structure label of the BPE sub-word unit are input into the text encoder in their original order to obtain the text conditional features.

[0012] Optionally, the acoustic model based on conditional flow matching is trained by repairing the audio and text conditional features to learn the mapping relationship from Gaussian noise distribution to Mel spectrum distribution, thereby obtaining a speech synthesis model, specifically including: An acoustic model based on conditional flow matching is constructed, and a linear interpolation path from the noise distribution to the target data distribution is preset. The linear interpolation path represents the transmission cost of the probability mass moving from the noise point to the data point along a straight line, and the time derivative of the linear interpolation path is used as the target vector field to be learned by the acoustic model. The text features of the text conditional features and the Mel spectrum features of the repaired audio are input into the acoustic model based on conditional flow matching to obtain the noise sample and target data at the current time step. The interpolation point at the current time step is determined along the linear interpolation path to obtain the prediction vector. With the goal of minimizing the difference between the predicted vector and the target vector field, the network parameters of the acoustic model based on conditional flow matching are updated through backpropagation to obtain the speech synthesis model.

[0013] This invention provides a Tibetan speech synthesis system, comprising: The data acquisition module is used to acquire the original ASR audio and its corresponding Tibetan text; The speech restoration module is used to perform end-to-end restoration processing on the original ASR audio through a neural network-based speech restoration model to repair damaged acoustic features and reconstruct high-fidelity time-domain waveforms to obtain restored audio. The text encoding module is used to encode Tibetan text byte-to-byte using the Xizang-Alpaca word segmenter to compress the original Tibetan character sequence into a sub-word sequence that retains syllable structure information, thereby obtaining multiple text features that characterize the sub-word features; and the G2P module is used to annotate the syllable structure corresponding to each text feature to provide syllable boundary information and pronunciation category differentiation annotations, thereby obtaining text conditional features; The model training module is used to train the conditional flow matching-based acoustic model by repairing audio and text conditional features to learn the mapping relationship from Gaussian noise distribution to Mel spectrum distribution, thereby obtaining a speech synthesis model. The speech synthesis module is used to input the target text and reference audio into the speech synthesis model to obtain Tibetan synthesized speech that matches the acoustic features of the target text and the reference audio.

[0014] This invention provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of a Tibetan speech synthesis method.

[0015] This invention provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of a Tibetan speech synthesis method.

[0016] The Tibetan speech synthesis method, system, device, and medium provided in this invention have the following advantages compared with the prior art: This invention uses the Xizang-Alpaca word segmenter to compress Tibetan text into a sequence of sub-words that preserves syllable structure information. Then, a G2P module is used to perform boundary labeling and pronunciation category differentiation for the syllables corresponding to each sub-word. This technique establishes a structured mapping between text features and acoustic features, enabling the conditional flow matching-based acoustic model to more accurately learn the unique duration distribution patterns, prosodic pauses, and coarticulation rules of Tibetan. This significantly improves the syllable clarity, rhythmic naturalness, and auditory naturalness of synthesized Tibetan speech, greatly enhancing its adaptability to personalized application scenarios. Attached Figure Description

[0017] Figure 1 Here is a flowchart of a Tibetan speech synthesis method provided in one embodiment; Figure 2 This is a schematic diagram of the Tibetan writing rules for a Tibetan speech synthesis method provided in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] This invention provides a Tibetan speech synthesis method, the method comprising: Obtain the original ASR audio and its corresponding Tibetan text.

[0020] The original ASR audio is repaired end-to-end using a neural network-based speech restoration model to restore damaged acoustic features and reconstruct high-fidelity time-domain waveforms, resulting in restored audio.

[0021] The Tibetan text is byte-pair encoded using the Xizang-Alpaca word segmenter to compress the original Tibetan character sequence into a sequence of subwords that retains syllable structure information, resulting in multiple text features representing the subword characteristics. The syllable structure corresponding to each text feature is then annotated using a G2P module to provide syllable boundary information and pronunciation category differentiation annotations, thus obtaining conditional text features.

[0022] By repairing the conditional features of audio and text, an acoustic model based on conditional flow matching is trained to learn the mapping relationship from Gaussian noise distribution to Mel spectrum distribution, thus obtaining a speech synthesis model.

[0023] The target text and reference audio are input into the speech synthesis model to obtain Tibetan synthesized speech that matches the acoustic features of the target text and the reference audio.

[0024] A specific embodiment of a Tibetan speech synthesis method is provided, such as... Figure 1 As shown, it includes: an ASR data migration pipeline based on neural repair, a word segmenter adaptation strategy for Tibetan, and acoustic model fine-tuning based on conditional flow matching.

[0025] 1. Data migration pipeline from ASR to TTS.

[0026] To transform noisy ASR data into high-quality samples suitable for TTS training, it is necessary to systematically address issues such as noise, reverberation, and missing frequency bands. Traditional spectral subtraction only processes the amplitude spectrum while ignoring the phase, easily introducing unnatural artifacts. To meet the high requirements of modern neural vocoders for phase consistency and spectral detail, this invention constructs a neural repair pipeline.

[0027] 1.1 Voice Fixer-based voice repair.

[0028] A pre-trained neural network-based voice restoration model (VoiceFixer) is used to perform end-to-end processing (inference mode, without retraining the model weights) on the original ASR audio. VoiceFixer is based on the ResUNet analysis network and the TFGAN vocoder, and adopts a two-stage strategy of analysis and synthesis.

[0029] Analysis Phase: VoiceFixer first recovers the original waveform for speech recognition. Its ResUNet-based analysis network utilizes cross-scale receptive fields and skip connections to nonlinearly model the damaged acoustic features of noise, reverberation, and other degradation factors in speech, resulting in a Mel spectrogram that more closely resembles clean speech.

[0030] Synthesis stage: The TFGAN vocoder reconstructs high-fidelity time-domain speech using adversarial waveforms based on the recovered Mel spectrogram. Subsequently, the recovered audio is uniformly resampled from 44.1 kHz to 24 kHz to match the input sampling rate required by the downstream TTS model.

[0031] VoiceFixer’s original training objectives include multi-resolution STFT loss, adversarial loss, feature matching loss, and Mel spectrum reconstruction loss, enabling the model to handle denoising, dereverberation, and bandwidth expansion tasks simultaneously.

[0032] 1.2 Quality screening based on SNR.

[0033] The process involves acquiring the speech segment and silence segment from each repaired audio file, and determining the first average energy of the speech segment and the second average energy of the silence segment. The signal-to-noise ratio (SNR) is determined based on the first and second average energies, and repaired audio files with an SNR below a preset threshold are discarded, resulting in multiple retained audio files. The root mean square amplitude (RMS) of each retained audio file is normalized to a target RMS amplitude using a preset overall gain coefficient, yielding preliminary speech. A pre-trained speech activity detection model is used to identify the speech boundaries of the preliminary speech, and initial and final silences are trimmed to obtain the final speech. The number of syllables per second in the final speech is compared with a preset syllable interval to discard final speech files that do not meet the preset syllable interval, resulting in the filtered repaired audio files.

[0034] Despite the significant improvement in neural repair, samples with extremely poor original quality still need to be removed. Let the average power of the speech segment be... The average power of background noise in the silent segment is The signal-to-noise ratio is defined as:

[0035] (1) (2) in, For the set of sampling points of the speech segment, This is the set of sampling points for the silent segment. This invention sets... A dB threshold is set, and samples below this value are discarded.

[0036] 1.3 RMS loudness normalization.

[0037] To ensure consistent loudness across different samples, each audio track is normalized to the target RMS amplitude (-20 dBFS). The length is... signal The RMS range is:

[0038] (3) Normalized gain and the normalized signal for: (4) 1.4 Mute processing and consistency verification.

[0039] A pre-trained Voice Activity Detection (VAD) model is used to identify speech boundaries in the input audio. The VAD is trained on speech data with speech / non-speech annotations, and learns the ability to distinguish between speech frames and non-speech frames using a speech activity classification loss. In application, the input audio is divided into consecutive frames and fed into the VAD. Speech and silence intervals are determined based on the output speech activity probabilities. Pre-set duration boundary buffers are retained at the beginning and end, initial and final silences are trimmed, and internal silence segments exceeding the preset duration are deleted. Finally, adjacent speech segments are concatenated to obtain a speech signal with well-defined boundaries.

[0040] set up To reserve margin at the edges, the reserved range is: (5) Regarding pauses in a sentence, if their duration Exceeding the maximum allowed value (This invention takes 0.5s), then the cutoff point is: (6) Finally, the word error rate (WER) of the audio and transcribed text is calculated to verify consistency, and a threshold is set. Remove mismatched samples: (7) in, , , These represent the number of replacements, deletions, and insertions, respectively. This is a reference word count.

[0041] 1.5 Syllable rate filtering.

[0042] After VAD cropping, the number of syllables per second in each speech segment is counted. The normal reading speed for Tibetan is generally 3-10 syllables per second. (Eliminate) or Abnormal samples were used to remove abnormal readings with extremely slow speaking speeds and overlapping or skipping recordings with excessively fast speaking speeds.

[0043] 2. Tibetan texts: standardization and G2P enhancement.

[0044] 2.1 Text normalization based on FST.

[0045] To address the significant differences between written and spoken Tibetan, a rule-driven text normalization scheme based on the Weighted Finite State Transformer (WFST) is adopted. The normalization process is formalized as a cascaded composition of a series of transformers:

[0046] (8) in, For input text, To normalize the text, , , , These rules correspond to the conversion rules for numbers, dates, symbols, and abbreviations, respectively. Through deterministic mapping, they effectively resolve ambiguities in the pronunciation of Tibetan numbers (e.g., whether "123" is read as "one hundred and twenty-three" or "one two three") and the standardized pronunciation of Sanskrit characters. The normalization module also outputs pause level labels and boundary labels for use by downstream acoustic models.

[0047] 2.2 BPE Compression and Information Entropy Optimization.

[0048] The normalized Tibetan text is initialized at the character level or byte level according to the original character order to form an initial symbol sequence. The pre-trained Xizang-Alpaca word segmenter is called, and the merging priority of adjacent symbol pairs is calculated according to the preset vocabulary and byte pair merging rules. Adjacent symbols are merged in turn according to the merging priority until there are no more adjacent symbol pairs that can be merged, resulting in a BPE sub-word sequence. The BPE sub-word sequence is mapped to the corresponding word number, and the correspondence between each word and the original Tibetan character and its corresponding syllable is preserved. The word number and its position encoding are input into the text encoder to generate text features for acoustic model condition control.

[0049] Specifically, the Tibetan text is split according to the original character order to obtain an initial symbol sequence. The byte pair merging rule table built into the Xizang-Alpaca word segmenter is retrieved; this table contains merging operation sequences sorted by priority. Adjacent symbol pairs are merged sequentially from high to low priority. After each merge, the current symbol sequence is updated, and the merging priority of adjacent symbol pairs is recalculated. This merging, updating, and calculation operation is repeated until no mergingable adjacent symbol pairs remain in the current symbol sequence, resulting in a BPE subword sequence. Each subword unit in the BPE subword sequence is mapped to a corresponding lexical number, and the correspondence between each lexical and the original Tibetan characters and their corresponding syllables in the original Tibetan text is preserved. The lexical numbers and their positional encodings are input into the text encoder to obtain multiple text features.

[0050] In one embodiment, the vocabulary size of the Xizang-Alpaca byte-pair encoding word segmenter is approximately 50,000; for Tibetan training corpora, the compression ratio of the BPE sub-word sequence relative to the length of the original character sequence is approximately 55%.

[0051] (9) in, Indicates the sequence length compression ratio. Represents the initial character sequence. This represents the sequence of subwords obtained after byte pair encoding.

[0052] 2.3 G2P enhancement for Tibetan syllable structure.

[0053] The BPE-segmented Tibetan text is syllable-segmented, and the superposition structure of each syllable is parsed according to the preset Tibetan syllable slot template. The base word, prefix, superscript, subscript, suffix and other component positions are identified. The correspondence between BPE token, original Tibetan characters, syllable units and phoneme sequences is established. Based on the identified syllable slots and their combination relationships, corresponding phoneme tags and structural tags are generated for each character or character position, and the character order, syllable boundaries and the pairing relationship between characters and phoneme slots are preserved.

[0054] For Tibetan syllables with the same base characters but different superposition structures, different G2P conditions are generated based on their slot combinations and structural labels to distinguish the corresponding pronunciation and tone variants. For out-of-vocabulary words, a preset rule-driven pronunciation backtracking module is invoked to deduce the corresponding phoneme sequence based on its character composition, syllable slot structure, and Tibetan spelling rules. The derivation result is then output as a G2P annotation, providing pronunciation conditions for words outside the vocabulary list and Sanskrit-to-Tibetan words. Finally, the BPEtoken representation, positional encoding, and position-by-position G2P structural labels are input into the text encoder in their original order to generate text conditional features for use by the acoustic model.

[0055] In this embodiment, the implementation process includes: for each BPE sub-word unit in the text features, reconstructing the Tibetan character segment corresponding to that BPE sub-word unit. The Tibetan character segment is segmented into syllables, and the superposition structure of each syllable is parsed according to a preset Tibetan syllable slot template. Using the base character, preceding characters, superscript characters, following characters, suffix characters, vowel symbols, and superscript symbols of the Tibetan character segment, structured labels describing the syllable slots and their combination relationships are obtained. Based on the syllable slots and their combination relationships in the structured labels, corresponding phoneme labels and G2P structure labels are generated for each character or character position, while preserving the character order, syllable boundaries, and the pairing relationship between characters and phoneme slots. For Tibetan syllables with the same base character but different superposition structures, different G2P conditions are generated based on their slot combinations and structure labels to distinguish the corresponding pronunciation and tone variants. For out-of-vocabulary words, a preset rule-driven pronunciation backtracking module is invoked to deduce the corresponding phoneme sequence based on its character composition, syllable slot structure, and Tibetan spelling rules. The derivation result is then used as a G2P annotation. The lexical number representation, positional encoding, and position-by-position G2P structure label of the BPE sub-word unit are input into the text encoder in their original order to obtain the text conditional features.

[0056] 3. Acoustic model based on conditional flow matching.

[0057] The CosyVoice2 architecture is adopted, and its core is conditional flow matching based on optimal transport (OT) (OT-CFM). The goal is to learn a time-varying... Changing vector field Gaussian noise distribution Transform into data distribution (Mel spectrum).

[0058] An acoustic model based on conditional flow matching is constructed, with a pre-defined linear interpolation path from the noise distribution to the target data distribution. The linear interpolation path represents the transmission cost of the probability mass moving from the noise point to the data point along a straight line, and the time derivative of the linear interpolation path is used as the target vector field to be learned by the acoustic model.

[0059] The text features of the text conditional features and the Mel spectrum features of the repaired audio are input into the acoustic model based on conditional flow matching to obtain the noise sample and target data at the current time step. The interpolation point at the current time step is determined along the linear interpolation path to obtain the prediction vector.

[0060] With the goal of minimizing the difference between the predicted vector and the target vector field, the network parameters of the acoustic model based on conditional flow matching are updated through backpropagation to obtain the speech synthesis model.

[0061] 3.1 Optimal transmission path.

[0062] Optimal transmission condition vector fields are employed to accelerate convergence and improve generation quality. Definitions are derived from noise. To target data linear interpolation path :

[0063] (10) The time derivative (target vector field) of this path is constant, indicating that the probability mass moves along a straight line from the noise point to the data point, minimizing the transmission cost. (11) 3.2 Training objectives.

[0064] Neural Networks Trained to fit the target vector field The loss function is defined as: (12) Among them, conditions Includes text encoding and reference audio style embedding .

[0065] 3.3 Classifier-free guidance (CFG).

[0066] To enhance control over voice prompts, classifier-free guidance (CFG) is introduced during inference. The modified vector field... for:

[0067] (13) Larger It can significantly improve the speaker similarity between synthesized speech and reference audio, and is used by default. .

[0068] 3.4 Zero-sample voice prompt mechanism.

[0069] Controllability is achieved through voice prompts during the reasoning phase. The user inputs the target text. and reference audio (3~10 seconds). The model is extracted by the encoder. The system uses global speaker vectors and local prosodic features, and the stream matching decoder uses these features as conditions to map text semantic tokens to Mel spectra. This enables the system to perform zero-sample cloning, generating speech with specific timbre, rate, and emotional tone without retraining.

[0070] 4. Experiment setup.

[0071] 4.1 Dataset and Preprocessing.

[0072] Table 1. Dataset Statistics Original ASR audio 2680.81 2481594 1200 16kHz Training set after cleaning 2054.4 705000 780 24kHz Validation set 114.9 39437 79 24kHz test set 14.2 4875 40 24kHz Listening test sample set — 100 10 24kHz The original data came from publicly available and collaboratively collected ASR corpus of the Tibetan Central Tibetan dialect, with an original duration of approximately 2680.81 hours. After cleaning, approximately 2054.4 hours of high-quality usable data were obtained. The cleaning retention rate was approximately 77%, with the main samples removed being low-quality recordings with an SNR < 15dB and misaligned samples with a WER > 10%. After FST normalization and BPE segmentation, the average sequence length was reduced from 87.3 at the character level to 39.5 at the BPE level, achieving a compression rate of approximately 54.7%. All audio was resampled to 24kHz.

[0073] 4.2 Baseline Model.

[0074] To ensure fairness in the comparison, all baseline models are trained on the same cleaned training set provided in this invention. While preserving the original text encoding method of each model design, the same text normalization output is used, an early stopping mechanism is added on the same validation set, and the models are evaluated under the same test set and evaluation protocol.

[0075] (1) Variational Inference with adversarial learning for end-to-end Text-to-Speech (VITS): A classic end-to-end model that combines VAE and normalized flow and uses Montreal Forced Aligner (MFA) for external forced alignment.

[0076] (2) Matcha-TTS: mainstream stream matching model, the baseline of competing products in the same paradigm, using OT-CFM but without voice prompt mechanism.

[0077] (3) Tibetan Multi-Dialect Text-to-Speech (TMD-TTS): It uses a dynamic routing mechanism to make dialect-specific fine-tuning and represents the current state-of-the-art (SOTA) in the Tibetan language field.

[0078] (4) Our invention: Based on the CosyVoice2-0.5B architecture, using the Xizang-Alpaca word segmenter (vocabulary size 50k), fine-tuning the HiFi-GAN vocoder on the transfer data, 20-step ODE solving, CFG scale. (See 5-second audio clip.)

[0079] All experiments were run on four NVIDIA RTX 4090 GPUs (24GB x 4 VRAM), with a batch size of 32 and a learning rate of [missing information]. Adam optimizer ( , Training for 50 epochs takes approximately 96 hours.

[0080] 4.3 Evaluation indicators.

[0081] Subjective indicators: (1) MOS (Mean Subjective Opinion Score): 20 native Tibetan speakers (10 males and 10 females, aged 20-45 years, from the U-Tsang region) were invited to rate the naturalness of the synthesized speech from 1 to 5 points (5 points being the best). 100 test sentences were randomly selected from each system, the sample order was randomized, and the listeners were independent of each other. The mean and 95% confidence interval (CI) were calculated, and the differences between paired systems were tested using the Wilcoxon signed-rank test (p<0.05 was considered significant). (2) Pronunciation accuracy (PA): The proportion of syllables in the synthesized samples that did not have stress, omission, or mispronunciation was counted.

[0082] Objective metrics: (1) WER (character error rate): The synthesized speech was identified using an independently trained Tibetan ASR evaluator (retrained for Tibetan based on Wenet, with no overlap between the training set and the training / test set of this study) to calculate the character error rate (lower is better). This evaluator is completely independent of the data-cleaned WER verification model to avoid cyclic bias. (2) MCD (Mel-Cepstral Distortion): Measures the distance between the synthesized speech and the spectral envelope of the real recording (lower is better, in dB). (3) DNSMOS (P.808 Perceptual Quality): The perceptual quality of the speech was predicted using the Microsoft DNSMOS pre-trained model (range 1~5, higher is better). Its reliability in Tibetan scenarios has been verified to be consistent with the MOS results (Pearson r=0.82). (4) SECS (Speaker Embedding Cosine Similarity): The x-vector cosine similarity between the synthesized speech and the reference prompt audio was calculated to evaluate the zero-sample timbre reproduction (higher is better). Only applicable to systems that support voice prompts. (5) RTF (Real-time Rate): Measured on a single NVIDIA RTX 4090 (batch size 1) the time required to generate a unit duration of speech (lower is better).

[0083] 5. Experimental Results and Analysis.

[0084] 5.1 Comparison of main results.

[0085] Table 2 Comparison of subjective and objective comprehensive evaluation results of Tibetan speech synthesis As shown in Table 2, in terms of naturalness, the MOS of this invention is 4.35, which is higher than VITS (3.92) and Matcha-TTS (4.15), and also higher than the TMD-TTS result listed in Table 2 (4.25). Since the 95% confidence intervals of Ours and TMD-TTS are close, this invention only uses the expression "significantly better than" when the Wilcoxon rank-sum test between the two systems reaches significance (p<0.05); for pairwise comparisons with small differences in MOS, the expression "higher than" is used.

[0086] In terms of intelligibility, CER decreased from 4.15% in VITS to 2.14% in this invention, a reduction of 48%. This is thanks to the introduction of the BPE tokenizer, which enables the model to capture Tibetan syllable-level phonetic information with more compact token sequences, fundamentally alleviating the attention alignment failure problem caused by excessive character-level fragmentation.

[0087] In terms of sound purity, the significant improvement in the DNSMOS score (4.12) demonstrates the effectiveness of the neural data cleaning pipeline, and the synthesized speech sounds significantly better than other baselines.

[0088] In terms of zero-sample controllability, the SECS reached 0.78, indicating that the model can reproduce the voice of a speaker who has not been seen with extremely high fidelity, which is an extremely high level in zero-sample generation under low resource conditions.

[0089] In terms of inference efficiency, the present invention has an RTF of 0.18, achieving a good balance between efficiency and quality in 20 ODE solving steps, meeting the requirements of real-time applications (RTF<1).

[0090] 5.2 Ablation experiment.

[0091] 5.2.1 Data preprocessing strategy ablation.

[0092] Table 3. Data migration pipeline components and text front-end ablation experiments As shown in Table 3, under the same CosyVoice 2 skeleton and BPE front end, the model trained with 2054.4 hours of migration data (configuration A) is slightly better than the version trained with 45 hours of recording data (configuration G) in both MOS and CER (MOS: 4.35 vs 4.32, CER: 2.14% vs 2.35%). This shows that the quantitative advantage of migration data can make up for the slight qualitative difference, and verifies the feasibility of the ASR-to-TTS data migration route.

[0093] Neural restoration outperforms traditional DSP (AvsB): After replacing VoiceFixer with spectral subtraction, MOS decreased by 0.60 points and DNSMOS decreased by 0.64 points, indicating that VoiceFixer's full-band integrated processing has a significant advantage in sound quality restoration.

[0094] Data cleaning is indispensable (AvsC): Using raw ASR data directly, the MOS is only 3.45, and the DNSMOS drops to 3.15. The background noise and mechanical feel are severe, which verifies that the "garbage in, garbage out" principle still applies.

[0095] The decisive role of the BPE segmenter (AvsD): When back to character-level input, CER surged to 15.4% (7.2 times), MOS dropped by 0.53 points, and the excessive length of character-level sequences caused serious attention alignment failures.

[0096] Synergistic gains of TN / G2P (AvsE, F): TN deficiency mainly affects non-standard words such as numbers / dates (CER increases by 1.75 percentage points, configuration E), while G2P enhancement mainly contributes to the stability of complex compound words and long sentences (CER increases by 0.39 percentage points, configuration F); together, they contribute approximately 2.14% to 2.53% of the intelligibility difference.

[0097] The necessity of multi-level screening (AvsH): When only SNR screening is performed and VAD, rate filtering and WER verification are skipped, MOS drops by 0.20 points and CER increases by 0.48 percentage points, indicating that the multi-level screening strategy can effectively remove samples with poor alignment and abnormal speech rate.

[0098] 5.2.2 ODE step ablation.

[0099] Table 4. Impact of ODE Inference Steps on Quality and Efficiency 5 4.10 2.89 0.05 10 4.28 2.31 0.09 20 (This invention) 4.35 2.14 0.18 50 4.36 2.12 0.43 20 steps represent the optimal trade-off between quality and efficiency: compared to 10 steps, MOS is improved by 0.07 and CER is reduced by 0.17 percentage points; compared to 50 steps, RTF is reduced by approximately 58% while the quality difference is minimal.

[0100] 5.2.3 CFG-guided scalar ablation.

[0101] Table 5. Influence of CFG guiding scale ω on naturalness and speaker similarity. 0 (No CFG) 3.95 0.52 1 4.15 0.65 3 (This invention) 4.35 0.78 5 4.28 0.83 7 4.11 0.87 As ω increases, speaker similarity monotonically increases, but MOS peaks near ω=3 and then decreases due to over-constraint. This invention selects ω=3 as the default value.

[0102] 5.2.4 Ablation of reference audio duration.

[0103] Table 6. Impact of Reference Audio Duration on Zero-Sample Controllability 1 second 4.12 0.58 3 seconds 4.28 0.72 5 seconds (this invention) 4.35 0.78 10 seconds 4.36 0.80 The impact of audio prompt duration on system performance is significant within the 35-second range, but tends to saturate above 5 seconds. In practical applications, it is recommended to use a 35-second reference audio to balance ease of use and effectiveness.

[0104] 5.3 Controllability analysis.

[0105] To verify the controllable generation capability based on voice prompts, audio prompts from different genders (8 men, 8 women), ages (20-60 years old), and speech rates (slow 3-4 syllables / second, standard 4.5-5.5 syllables / second, and fast 6-8 syllables / second) were selected for testing. The experiment found that:

[0106] The model can reproduce the voice of an unseen speaker with extremely high fidelity, with an average SECS of 0.78 and a maximum of 0.91.

[0107] Spectral analysis showed that the F0 distribution of the synthesized speech was highly consistent with that of the reference audio (Pearson correlation coefficient r=0.86).

[0108] When using fast-paced audio prompts, the syllable rate of the synthesized speech is automatically adjusted to approximately 6.5 syllables / second, while the standard prompt corresponds to 4.5 syllables / second. This verifies that the stream matching model achieves decoupling and reconstruction of prosodic style in the latent space.

[0109] The SECS of the transgender cue test (male text + female cue) was 0.71, slightly lower than that of the same-gender test (0.80), but still maintained the distinctiveness of the timbre.

[0110] Based on the same inventive concept, the present invention also provides a Tibetan speech synthesis system, the system comprising: The data acquisition module is used to acquire the original ASR audio and its corresponding Tibetan text.

[0111] The speech restoration module is used to perform end-to-end restoration processing on the original ASR audio using a neural network-based speech restoration model to repair damaged acoustic features and reconstruct high-fidelity time-domain waveforms to obtain restored audio.

[0112] The text encoding module is used to encode Tibetan text byte-to-byte using the Xizang-Alpaca word segmenter, compressing the original Tibetan character sequence into a sequence of subwords that retains syllable structure information, resulting in multiple text features representing the subword characteristics. The G2P module then annotates the syllable structure corresponding to each text feature to provide syllable boundary information and pronunciation category differentiation annotations, thus obtaining conditional text features.

[0113] The model training module is used to train the conditional flow matching-based acoustic model by repairing audio and text conditional features to learn the mapping relationship from Gaussian noise distribution to Mel spectrum distribution, thereby obtaining a speech synthesis model.

[0114] The speech synthesis module is used to input the target text and reference audio into the speech synthesis model to obtain Tibetan synthesized speech that matches the acoustic features of the target text and the reference audio.

[0115] Based on the same inventive concept, embodiments of the present invention also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a Tibetan speech synthesis method.

[0116] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of a Tibetan speech synthesis method.

[0117] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for synthesizing Tibetan speech, characterized in that, include: Obtain the original ASR audio and its corresponding Tibetan text; The original ASR audio is repaired end-to-end using a neural network-based speech restoration model to restore damaged acoustic features and reconstruct high-fidelity time-domain waveforms, resulting in restored audio. The Tibetan text is byte-pair encoded using the Xizang-Alpaca word segmenter to compress the original Tibetan character sequence into a sub-word sequence that retains syllable structure information, thereby obtaining multiple text features that characterize the sub-word features. The G2P module is used to annotate the syllable structure corresponding to each text feature to provide syllable boundary information and pronunciation category differentiation annotation, thus obtaining text conditional features; The acoustic model based on conditional flow matching is trained using the repaired audio and the conditional text features to learn the mapping relationship from Gaussian noise distribution to Mel spectrum distribution, thereby obtaining a speech synthesis model. The target text and reference audio are input into the speech synthesis model to obtain Tibetan synthesized speech that matches the target text and the acoustic features of the reference audio.

2. The Tibetan speech synthesis method as described in claim 1, characterized in that, The process involves end-to-end restoration of the original ASR audio using a neural network-based speech restoration model to repair damaged acoustic features and reconstruct high-fidelity time-domain waveforms, resulting in restored audio. Specifically, this includes: The neural network-based speech restoration model includes: a ResUNet analysis network and a TFGAN vocoder; By using the deep convolutional feature extraction and skip connection mechanism of the cross-scale receptive field of the ResUNet analysis network, the damaged acoustic features are nonlinearly modeled to obtain a Mel spectrogram that is closer to the pure speech. Based on the Mel spectrogram, high-fidelity time-domain speech is reconstructed using the TFGAN vocoder with adversarial waveforms to obtain the restored audio.

3. The Tibetan speech synthesis method as described in claim 2, characterized in that, After obtaining the repaired audio, the process also includes multi-dimensional filtering of the repaired audio, specifically including: Obtain the speech segment and silence segment in each repaired audio, and determine the first average energy of the speech segment and the second average energy of the silence segment respectively; The signal-to-noise ratio is determined based on the first average energy and the second average energy, and the repaired audio corresponding to the signal-to-noise ratio being lower than a preset threshold is removed to obtain multiple retained audio; The root mean square amplitude of each preserved audio track is normalized to the target root mean square amplitude based on the preset overall gain coefficient, thus obtaining the preliminary speech. The speech boundaries of the preliminary speech are identified by a pre-trained speech activity detection model, and the beginning and end silences are trimmed to obtain the final speech. The number of syllables per second in the final language is compared with the preset syllable interval to eliminate the final speech that does not meet the preset syllable interval, thus obtaining the filtered and repaired audio.

4. The Tibetan speech synthesis method as described in claim 1, characterized in that, Before the Tibetan text is encoded byte-by-byte using the Xizang-Alpaca word segmenter, the text undergoes normalization processing, specifically including: Constructing a text normalization cascade converter based on a weighted finite state converter; The input text is deterministically mapped by a text normalization cascade converter through number conversion rules, date conversion rules, symbol conversion rules, and abbreviation conversion rules to obtain normalized Tibetan text, and pause level labels and boundary labels are output simultaneously.

5. The Tibetan speech synthesis method as described in claim 1, characterized in that, The method involves using a Xizang-Alpaca word segmenter to encode the Tibetan text byte-to-byte, compressing the original Tibetan character sequence into a sub-word sequence that preserves syllable structure information, thereby obtaining multiple text features representing sub-word characteristics, specifically including: The Tibetan text is split according to the original character order to obtain an initial symbol sequence; Obtain the byte pair merging rule table built into the Xizang-Alpaca word segmenter, which contains a sequence of merging operations sorted by priority; Adjacent symbol pairs are merged in descending order of merging priority. After each merge, the current symbol sequence is updated and the merging priority of adjacent symbol pairs is recalculated. The merging, updating and counting operations are repeated. Merging stops when there are no adjacent symbol pairs that can be merged in the current symbol sequence, resulting in the BPE subword sequence. Each sub-word unit in the BPE sub-word sequence is mapped to a corresponding lexical number, and the correspondence between each lexical and the original Tibetan characters and their corresponding syllables in the original Tibetan text is preserved; The word numbers and their position codes are input into the text encoder to obtain multiple text features.

6. The Tibetan speech synthesis method as described in claim 5, characterized in that, The G2P module is used to annotate the syllable structure corresponding to each text feature to provide syllable boundary information and pronunciation category differentiation annotations, thereby obtaining text conditional features, specifically including: For each BPE sub-word unit in the text feature, the Tibetan character fragment corresponding to that BPE sub-word unit is restored; The Tibetan character fragments are segmented into syllables, and the superposition structure of each syllable is parsed according to the preset Tibetan syllable slot template. The base character, preceding character, superscript character, subscript character, suffix character, subsequent superscript character, vowel symbol, and superscript character symbol of the Tibetan character fragment are used to obtain structured labels describing syllable slots and their combination relationships. Based on the syllable slots and their combination relationships in the structured tags, corresponding phoneme tags and G2P structure tags are generated for each character or character position, while preserving the character order, syllable boundaries, and pairing relationships between characters and phoneme slots; For Tibetan syllables with the same base word but different superposition structures, different G2P conditions are generated based on their slot combinations and structural tags to distinguish the corresponding pronunciation and tone variants. For words not registered in the database, the preset rules drive the pronunciation backtracking module to deduce the corresponding phoneme sequence based on its character composition, syllable slot structure and Tibetan spelling rules, and use the derivation result as a G2P annotation. The lexical number representation, positional encoding, and position-by-position G2P structure label of the BPE sub-word unit are input into the text encoder in their original order to obtain the text conditional features.

7. The Tibetan speech synthesis method as described in claim 1, characterized in that, The process of training the conditional flow matching-based acoustic model using the repaired audio and the text conditional features to learn the mapping relationship from Gaussian noise distribution to Mel spectrum distribution, thereby obtaining a speech synthesis model, specifically includes: An acoustic model based on conditional flow matching is constructed, and a linear interpolation path from the noise distribution to the target data distribution is preset. The linear interpolation path represents the transmission cost of the probability mass moving from the noise point to the data point along a straight line, and the time derivative of the linear interpolation path is used as the target vector field to be learned by the acoustic model. The text features of the text conditional features and the Mel spectrum features of the repaired audio are input into the acoustic model based on conditional flow matching to obtain the noise sample and target data at the current time step. The interpolation point at the current time step is determined along the linear interpolation path to obtain the prediction vector. With the goal of minimizing the difference between the predicted vector and the target vector field, the network parameters of the acoustic model based on conditional flow matching are updated through backpropagation to obtain the speech synthesis model.

8. A Tibetan speech synthesis system, characterized in that, include: The data acquisition module is used to acquire the original ASR audio and its corresponding Tibetan text; The speech restoration module is used to perform end-to-end restoration processing on the original ASR audio through a neural network-based speech restoration model to repair damaged acoustic features and reconstruct high-fidelity time-domain waveforms to obtain restored audio. The text encoding module is used to encode the Tibetan text byte-to-byte using the Xizang-Alpaca word segmenter, so as to compress the original Tibetan character sequence into a sub-word sequence that retains syllable structure information, and obtain multiple text features representing sub-word features; The G2P module is used to annotate the syllable structure corresponding to each text feature to provide syllable boundary information and pronunciation category differentiation annotation, thus obtaining text conditional features; The model training module is used to train the acoustic model based on conditional flow matching using the repaired audio and the text conditional features to learn the mapping relationship from Gaussian noise distribution to Mel spectrum distribution, thereby obtaining a speech synthesis model. The speech synthesis module is used to input the target text and reference audio into the speech synthesis model to obtain Tibetan synthesized speech that conforms to the target text and matches the acoustic features of the reference audio.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the Tibetan speech synthesis method according to any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the Tibetan speech synthesis method according to any one of claims 1-7.

Images