Tibetan language Andor dialect speech synthesis method
By using the DIT end-to-end architecture and full-scale fine-tuning algorithm, combined with multilingual data pre-training and Tibetan data fine-tuning, the problem of insufficient data in the speech synthesis of the Amdo dialect of Tibetan was solved, and high-quality, natural and stable speech synthesis results were achieved.
Patent Information
- Application Number
- CN202512047296.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-03-17
AI Technical Summary
Existing technologies suffer from insufficient data resources in the synthesis of Amdo Tibetan dialect, resulting in low naturalness of synthesized speech, a lack of fluency and realism reminiscent of human speech, and unstable model training, which fails to fully learn acoustic features and prosodic rules.
A text-to-speech model is constructed using a DIT end-to-end architecture. It is pre-trained using a multilingual text-audio pairing dataset and combined with a full fine-tuning algorithm. The diffusion decoder is trained through cross-language transfer learning and conditional flow matching (CFM) to gradually generate high-quality Mel spectrograms. Finally, it is converted into audio waveforms through a neural vocoder.
It achieves high-quality, natural and stable speech synthesis of the Amdo dialect of Tibetan, reduces the complexity and cost of data collection, enhances the model's adaptability to the Tibetan language system, and significantly improves the naturalness and realism of the generated speech.
Smart Images
Figure CN121687005A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of speech synthesis technology, specifically relating to a method and system for speech synthesis of the Amdo dialect of Tibetan. Background Technology
[0002] Text-to-Speech (TTS) technology aims to automatically convert text information into natural and fluent human speech. It is one of the core technologies in fields such as human-computer interaction, intelligent assistants, audio reading, accessibility assistance, language education, and intelligent customer service.
[0003] Early speech synthesis techniques were primarily based on rule-based and parametric methods, including formant synthesis, linear predictive coding (LPC) synthesis, and waveform concatenation synthesis. These methods relied on rules and parameters hand-designed by phonetics experts. While they could generate understandable speech in specific scenarios, the synthesis quality was generally poor, exhibiting issues such as harsh sound quality, a mechanical feel, unnatural rhythm, and a lack of personalized expression. In the 21st century, statistical parametric speech synthesis methods gradually became mainstream, with Hidden Markov Models (HMMs) being a widely used approach. These methods learn the mapping relationship between text features and acoustic parameters through statistical modeling, generating relatively smooth and coherent speech. However, HMM-synthesized speech often suffers from oversmoothing, resulting in a blurry and unclear sound. With the rapid development of deep learning technology, especially the widespread application of end-to-end neural network models, speech synthesis technology has achieved a revolutionary breakthrough. Based on deep neural networks, traditional methods have broken free from their dependence on complex phonetics knowledge and hand-designed features, enabling the direct learning of end-to-end mapping relationships from large-scale text-audio pairing data. However, these technological achievements are mainly concentrated on mainstream languages with abundant data resources. For some low-resource languages, the use of deep neural networks for speech synthesis still has the limitation of low generation quality.
[0004] Tibetan is one of the important minority languages in China, with a long history and rich cultural connotations. Amdo dialect, one of the three major Tibetan dialects, is mainly distributed in Qinghai, Gansu, and Sichuan provinces, and has a large number of speakers. As a low-resource language, Amdo Tibetan lacks large-scale, high-quality labeled speech data, leading to problems such as high complexity and unstable training models, hindering the learning of rich acoustic features and prosodic patterns. Furthermore, parameter freezing strategies result in rigid prosodic representations and limited acoustic feature modeling, preventing sufficient adjustment of deep model representations to adapt to the characteristics of the Tibetan language. This results in synthesized speech with extremely low naturalness, sounding mechanical and monotonous, lacking the fluency and realism of human speech. Summary of the Invention
[0005] To address the shortcomings of existing technologies in the speech synthesis process of the Amdo dialect of Tibetan, this invention provides a method and system for speech synthesis of the Amdo dialect of Tibetan.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for speech synthesis of the Amdo dialect of Tibetan includes the following steps: Obtain a multilingual text-audio pairing dataset; construct a text-to-speech model based on the DIT end-to-end architecture; train the text-to-speech model using the data in the multilingual text-audio pairing dataset to obtain a multilingual general model; Obtain text-audio pairing data of the Amdo dialect of Tibetan, input the text-audio pairing data of the Amdo dialect of Tibetan into a multilingual general model, and update the model parameters of the multilingual general model using a full fine-tuning algorithm; Obtain the text of the Amdo dialect of Tibetan to be identified, input the text of the Amdo dialect of Tibetan to be identified into the multilingual general model with updated parameters, and obtain the synthesized speech of the Amdo dialect of Tibetan.
[0007] Preferably, the text-to-speech model includes a text encoder, a duration predictor, a diffusion decoder, and a vocoder. When training the text-to-speech model using data from the multilingual text-audio pairing dataset, specifically, the text encoder and duration predictor are trained using a loss function; the diffusion decoder is trained using conditional flow matching (CFM); and the pre-trained model is optimized by weighting the losses of the text encoder, duration predictor, and diffusion decoder to obtain a multilingual universal model.
[0008] Preferably, the diffusion decoder is trained using Conditional Flow Matching (CFM). Specifically, the real Mel spectrogram is used as the target data in the diffusion decoder; different levels of Gaussian noise are added to the target data to construct a diffusion path from clear data to pure noise; the model learns to predict the velocity field from the noise state to the clear spectrum; the optimal transmission path from the noise distribution to the target spectrum distribution is learned using text features as conditions; and based on the optimal transmission path, a Mel spectrogram aligned with the input text is gradually generated through a multi-step iterative denoising process.
[0009] Preferably, the total loss is calculated using the following formula: ; in, For conditional flow matching loss; Predict losses based on duration; For pitch prediction loss.
[0010] Preferably, updating the model parameters of the multilingual universal model using the full fine-tuning algorithm specifically includes the following steps: Use all the model parameters of the multilingual general model obtained in the pre-training stage as the initial parameter values for fine-tuning; Iterative training based on the initial parameter values specifically includes: randomly selecting a batch of training samples, each batch including text-audio pairing data of the Amdo dialect of Tibetan; inputting the text into a text encoder to obtain a high-dimensional feature representation of the text; then inputting the text features into a duration predictor to predict the pronunciation duration of each character, and comparing it with the actual duration to calculate the duration prediction loss; performing upsampling alignment on the text features according to the predicted duration to match the number of audio frames; then training the diffusion model, specifically including: randomly sampling time steps, generating corresponding noise vectors, calculating the interpolation state between the noise and the target spectrum, inputting the interpolation state into the diffusion decoder to predict the velocity field, calculating the conditional flow matching loss between the predicted velocity field and the actual velocity field, and calculating the total loss under Tibetan; updating the model parameters of the multilingual general model based on the total loss under Tibetan.
[0011] Preferably, the step of obtaining the Amdo dialect text of Tibetan to be identified, and inputting the Amdo dialect text of Tibetan to be identified into the multilingual general model with updated parameters to obtain synthesized speech of Amdo dialect of Tibetan, specifically includes the following steps: The input Tibetan Amdo dialect text to be recognized is preprocessed; The preprocessed text is input into a finely tuned text encoder, which converts the text character sequence into a high-dimensional feature vector representation to obtain text features. The text features are input into a finely tuned duration predictor, which outputs the corresponding pronunciation duration for each text character. Based on the predicted duration information, the text features are upsampled and expanded to match the length of the text feature sequence with the number of Mel spectrum frames to be generated. The length of the expanded features is equal to the sum of the predicted durations of all characters. Initialize a Gaussian noise matrix with the same shape as the target Mel spectrum, and perform iterative denoising using differential equation solving methods to obtain a clear Mel spectrum. The generated Mel spectrogram is input into a neural vocoder to obtain an audio waveform signal, resulting in high-quality synthesized speech of the Amdo dialect of Tibetan.
[0012] Preferably, the method further includes preprocessing the text-audio pairing data of the Amdo dialect of Tibetan before inputting the text-audio pairing data into the multilingual universal model. The text-audio pairing data includes audio data and text data, specifically including the following steps: The audio data undergoes noise reduction, sentence segmentation, silence trimming, and resampling; the text data undergoes unified encoding, number conversion, punctuation processing, special marker addition, and syllable segmentation. Mel spectrograms are extracted from the preprocessed audio data; Tibetan character-level or syllable-level word segmenters are used to convert the preprocessed text data into character sequences that the model can process. The ConvNeXT module is introduced to further process the text features. The reprocessed text data is then aligned with the audio data to obtain training data pairs for fine-tuning.
[0013] This invention also provides a Tibetan Amdo dialect speech synthesis system, specifically comprising: Model construction is used to obtain a multilingual text-audio pairing dataset; a text-to-speech model is built based on the DIT end-to-end architecture; the text-to-speech model is trained using the data in the multilingual text-audio pairing dataset to obtain a multilingual general model.
[0014] The parameter tuning module is used to acquire text-audio pairing data of the Amdo dialect of Tibetan, input the text-audio pairing data of the Amdo dialect of Tibetan into the multilingual general model, and update the model parameters of the multilingual general model using the full fine-tuning algorithm.
[0015] The speech synthesis module is used to acquire the text of the Amdo dialect of Tibetan to be identified, and to obtain the synthesized speech of the Amdo dialect of Tibetan by updating the input parameters of the text of the Amdo dialect of Tibetan to be identified into a multilingual general model.
[0016] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps described in the method for synthesizing the speech of the Amdo dialect of Tibetan.
[0017] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute the steps described in the method for synthesizing the speech of the Amdo dialect of Tibetan.
[0018] The Tibetan Amdo dialect speech synthesis method provided by this invention has the following beneficial effects: This invention comprehensively utilizes large-scale multilingual text-audio pairing datasets and small-scale text-audio pairing data of the Amdo Tibetan dialect, eliminating the need for special intermediate corpora and reducing the complexity and cost of data collection. Through cross-lingual transfer learning techniques, it fully leverages the acoustic features and prosodic rules learned by general models pre-trained on resource-rich languages (such as Chinese and English). This general knowledge is then transferred to the Amdo Tibetan dialect. A full-scale fine-tuning strategy is employed, directly updating model parameters using Tibetan data on top of the pre-trained model, avoiding parameter freezing and enhancing the model's adaptability to the Tibetan language system. Furthermore, the full-scale fine-tuning strategy allows all parameters to be updated during the fine-tuning phase, improving the model's processing flexibility and resulting in high-quality, natural, and stable Amdo Tibetan speech. Attached Figure Description
[0019] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating the architecture of a Tibetan Amdo dialect speech synthesis method according to an embodiment of the present invention.
[0021] Figure 2 This is a flowchart of a Tibetan Amdo dialect speech synthesis method according to the present invention. Detailed Implementation
[0022] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.
[0023] Example This invention provides a method for speech synthesis of the Amdo dialect of Tibetan, such as... Figure 1 As shown, it includes the pre-training stage, the full fine-tuning stage, and the inference stage.
[0024] (a) Pre-training phase Step 1: Construct a large-scale multilingual speech pre-training dataset. Large-scale multilingual pre-training data is obtained based on publicly available datasets, along with small-scale fine-tuning data for the Amdo Tibetan dialect. The specific composition is shown in Table 1.
[0025] Table 1 Pre-training dataset These data cover different language types, pronunciation methods, and prosodic patterns, providing the model with rich prior knowledge of speech. In particular, the inclusion of Chinese data is especially important for Tibetan, as both belong to the Sino-Tibetan language family and share certain commonalities in acoustics and prosody.
[0026] Step 2: Build a pre-trained model based on the end-to-end architecture of Diffusion Transformer (DiT), which includes four core modules: text encoder, duration predictor, diffusion decoder, and vocoder.
[0027] A text encoder converts an input sequence of text characters into a high-dimensional hidden representation. Employing a standard Transformer Encoder architecture, the input text first passes through a character embedding layer to be converted into a vector representation. Then, positional encoding is added to preserve sequence order information. Finally, the high-dimensional hidden representation of the text is obtained through multiple Transformer layers. The encoder consists of multiple stacked Transformer layers (typically 6-12 layers). Each layer includes: a multi-head self-attention mechanism to capture long-range dependencies in the text sequence; a feed-forward network for non-linear feature transformation; layer normalization; and residual connections to stabilize the training process.
[0028] The Duration Predictor is responsible for predicting the pronunciation duration of each text character (or phoneme), in units of audio frames. This module employs a lightweight convolutional neural network architecture, containing 2-3 one-dimensional convolutional layers. During training, the duration distribution patterns are learned from real text-audio alignment data through supervised learning. During inference, the predicted durations are used to upsampling the text features, matching their length to the number of Mel spectrum frames to be generated.
[0029] Unlike existing schemes that use conditional flow matching (CFM) as a local branch module, the diffusion decoder uses CFM as the core training paradigm for the entire diffusion model, achieving unified end-to-end generation from noise to spectrum.
[0030] The diffusion decoder employs a Diffusion Transformer (DiT) architecture, which generates high-quality Mel spectra from random noise through a progressive denoising process. Conditional flow matching is integrated throughout the entire diffusion generation process; it is not merely a function of a single module, but rather the overall training and inference paradigm of the model.
[0031] Unlike existing schemes that use Conditional Flow Matching (CFM) as a local branch module, the Diffusion Decoder in this invention uses CFM as the core training paradigm for the entire Diffusion Decoder, achieving unified end-to-end generation from noise to spectrum. The Diffusion Decoder employs a Diffusion Transformer (DiT) architecture, generating high-quality Mel spectra from random noise through a progressive denoising process. CFM is specifically used to train the Diffusion Decoder module, permeating its entire training and inference process. During training, the Diffusion Decoder is optimized using CFM loss; while the text encoder and duration predictor are trained using their respective loss functions, ultimately achieving joint optimization of the entire model through a weighted total loss.
[0032] During the training phase: a) Use real Mel spectrograms as target data; b) Add Gaussian noise of varying degrees to the target data to construct a diffusion path from clear data to pure noise; c) The model learns to predict the "velocity field" from the noisy state to the clear spectrum. d) Conditional Flow Matching (CFM) is used as the end-to-end training paradigm for the diffusion decoder. The optimal transmission path from the noise distribution to the target spectral distribution is learned by using text features as conditions.
[0033] During the reasoning stage: a) Start with random Gaussian noise; b) Based on the optimal transmission path learned by CFM, a clear Mel spectrum is gradually generated through a multi-step iterative denoising process (ODE solution); c) At each step, guided by the textual conditions, the movement follows the direction of the velocity field predicted by the CFM to ensure that the generated spectrum is aligned with the input text; The vocoder converts the generated Mel spectrogram into the final audio waveform. This invention employs a high-quality neural vocoder such as HiFi-GAN or Vocos. Vocoders are typically pre-trained on large-scale data and can be used directly without retraining for Tibetan.
[0034] Step 3: Perform end-to-end training on the pre-trained model. Learn the optimal transmission path from the noise distribution to the target data distribution, enabling the entire model to iteratively generate high-quality spectrum from random noise through ODE solving.
[0035] (1) Noise sampling: Generate a random Gaussian noise vector x0 with the same shape as the target spectrum x1, x0 ~ N(0,I).
[0036] (2) Path interpolation: At time step t (uniform sampling from 0 to 1), an intermediate state is constructed by linear interpolation: x = (1-t)·x0 + t·x1; where t = 0 is pure noise and t = 1 is the target spectrum.
[0037] (3) Velocity field prediction: The intermediate state x t Input the time step t and text condition c into the diffusion decoder to predict the velocity field at that point (i.e., the direction and velocity of movement from x0 to x1): v_pred=DiT(x,t,c).
[0038] (4) Loss calculation: The true velocity field is: v_true=x1-x0.
[0039] The mean squared error between the model-predicted velocity field and the actual velocity field is used as the loss function: L_CFM=||v_pred-v_true||².
[0040] (5) Parameter update: Update the model parameters θ through the backpropagation algorithm.
[0041] After pre-training, a multilingual general model was obtained, which has a powerful cross-language speech generation capability, laying a solid foundation for subsequent Tibetan fine-tuning.
[0042] (II) Full-scale fine-tuning stage The pre-trained model was fine-tuned using a small amount of Amdo dialect data (5-20 hours) to quickly adapt it to the unique phonetic features of Tibetan.
[0043] Step 1: Construct a high-quality Tibetan Amdo dialect fine-tuning dataset as training text. It must meet the technical specifications shown in Table 2.
[0044] Table 2 Technical Specifications of the Tibetan Language Fine-tuning Dataset Step 2: Preprocess the data in the high-quality Tibetan Amdo dialect fine-tuning dataset. Preprocessing of the Tibetan data is crucial for successful fine-tuning and includes six steps: audio preprocessing, text preprocessing, feature extraction, text encoding, time alignment, and contextual feature enhancement.
[0045] (1) Audio preprocessing, specifically including the following processes: Noise reduction: Use spectral subtraction or deep learning noise reduction models to remove background noise; Volume normalization: Normalize audio peaks to -1dB to ensure volume consistency; Sentence segmentation: Segment long audio into independent sentence segments according to natural pauses; Silence trimming: Remove excessively long silence segments at the beginning and end of sentences, retaining natural pauses of 50-100ms; Resampling: Unify the sampling rate to 22050Hz or 24000Hz.
[0046] (2) Preprocessing of Tibetan text, including: Unified encoding: ensure that all Tibetan text uses UTF-8 encoding; Number conversion: convert Arabic numerals to Tibetan pronunciation (e.g., "2024" → "ཉིས་སྟོང་ཉི་ཤུ་བཞི"); Punctuation processing: convert Tibetan punctuation marks to pause marks (e.g., "།" is marked as [PAUSE]); Addition of special marks: insert breathing mark [BREATH], long pause mark [LONG_PAUSE], etc.; Syllable segmentation: segment Tibetan text into syllables to facilitate subsequent encoding.
[0047] (3) Feature extraction. Mel-spectrum is extracted from the preprocessed audio as the training target, and the short-time Fourier transform (STFT) parameters and Mel filter bank parameters are set.
[0048] Short-Time Fourier Transform (STFT) parameters: FFT window length: 1024 or 2048 sampling points; Hop Length: 256 sampling points; Window function: Hann window.
[0049] Mel filter bank: Mel band count: 80 dimensions; frequency range: 0-8000 Hz; Mel spectrum with logarithmic scale.
[0050] The output is a two-dimensional matrix of shape [T, 80], where T is the number of time frames and 80 is the Mel band dimension.
[0051] (4) Text encoding: Tibetan character-level or syllable-level word segmenters are used to convert the preprocessed Tibetan text into a character sequence that the model can process.
[0052] Vocabulary construction: <pad>:0 (fill symbol); <bos>:1 (sentence-initial marker); <eos>:2 (sentence-end marker); <unk>:3 (Unknown character); [PAUSE]:4 (Pause marker); [BREATH]:5 (Breath marker). Tibetan characters:6-N (Tibetan syllables or characters, N is usually 500-1000).
[0053] Encoded output: an integer sequence of length L, where L is the number of characters.
[0054] (5) Time alignment and padding: To ensure that the text sequence matches the number of Mel spectrum frames, time alignment is required. Forced alignment: Use a pre-trained alignment model (such as Montreal Forced Aligner) to obtain the start and end times of each character; Filler tokens: Insert special filler tokens between characters to make the length of the text sequence equal to the number of spectrum frames T.
[0055] (6) Contextual feature enhancement: To better integrate text and audio features, the ConvNeXT module is introduced to further process text features, including: The character embedding vector is input into the ConvNeXT layer. The ConvNeXT layer contains: depthwise convolution with a kernel size of 7; layer normalization; linear expansion layer (expanding the dimension by 4 times); GELU activation function; linear projection layer (restoring the original dimension); and residual connections.
[0056] The output is enhanced text features, which are better suited for alignment with audio features.
[0057] After the above six steps, training data pairs (text features, Mel spectrum) are obtained for fine-tuning.
[0058] Step 3: Update all parameters of the multilingual general model through full fine-tuning to achieve deep adaptation to the Amdo dialect of Tibetan. The core challenge of full fine-tuning is "catastrophic forgetting": the model may forget the general knowledge learned in the pre-training stage when learning Tibetan features. Technical strategies to prevent catastrophic forgetting are shown in Table 3.
[0059] Table 3 Technical Strategies for Preventing Catastrophic Forgetting The specific algorithm processing steps for full-scale fine-tuning are as follows: The first step is initialization preparation: all parameters of the multilingual general model obtained in the pre-training stage are used as the initial parameter values for fine-tuning to ensure that the model starts learning Tibetan features from a state with rich speech knowledge.
[0060] The second step is to execute the main training loop, performing the following operations sequentially in each training iteration: (1) Data sampling: A batch of training samples was randomly selected from the preprocessed Tibetan Amdo dialect dataset, and each batch contained B text-audio pairing data; (2) Forward propagation calculation of various losses: First, the text is input into the text encoder to obtain the high-dimensional feature representation of the text; then the text features are input into the duration predictor to predict the pronunciation duration of each character and compare it with the actual duration to calculate the duration prediction loss; then the text features are upsampled and aligned according to the predicted duration to match the number of audio frames; then the diffusion model is trained, which includes: random sampling time steps, generating the corresponding noise vector, calculating the interpolation state between the noise and the target spectrum, inputting the interpolation state into the diffusion decoder to predict the velocity field, and calculating the conditional flow matching loss between the predicted velocity field and the actual velocity field; finally, the conditional flow matching loss, duration prediction loss and optional pitch prediction loss are weighted and summed according to the weight coefficients to obtain the total loss. The weights are configured as follows: conditional flow matching loss weight 1.0, duration loss weight 0.1 and pitch loss weight 0.05. (3) Backpropagation and parameter update: Based on the calculated total loss, the gradients of all model parameters are calculated through the backpropagation algorithm, and then the optimizer is used to update all parameters of the model; (4) Dynamic adjustment of learning rate: The learning rate scheduling strategy combining warm-up and cosine annealing is adopted. In the first 1000 steps of training, the learning rate increases linearly from zero to the maximum value, and then gradually decays to the minimum value according to the cosine function curve to ensure that the training process is stable and fully converged. (5) Early stopping mechanism: Continuously monitor the loss change on the validation set. If the validation loss does not decrease for three consecutive evaluation periods, the training will be terminated early to prevent the model from overfitting and maintain good generalization ability.
[0061] Speech synthesis algorithms specifically include: (1) Text preprocessing stage: The input Tibetan Amdo dialect text is standardized, including unifying the character encoding format, converting numbers into Tibetan pronunciation, and processing punctuation marks; then, word segmentation encoding is performed according to Tibetan syllables or characters; finally, special symbols such as sentence beginning marks and sentence ending marks are added at the beginning and end of the text sequence to complete the text preprocessing.
[0062] (2) Text encoding stage: The preprocessed text is input into the fine-tuned text encoder. The encoder converts the text character sequence into a high-dimensional feature vector representation. The output text feature matrix dimension is the text length multiplied by the hidden layer dimension.
[0063] (3) Duration prediction stage: Input the text features into the fine-tuned duration predictor. The predictor outputs the corresponding pronunciation duration for each text character. The duration is in the number of audio frames and the output is a positive integer sequence.
[0064] (4) Upsampling and alignment stage: Based on the predicted duration information, the text features are upsampled and expanded to match the length of the text feature sequence with the number of Mel spectrum frames to be generated. The length of the expanded features is equal to the sum of the predicted durations of all characters.
[0065] (5) Diffusion sampling to generate Mel spectrum stage: First, initialize a Gaussian noise matrix with the same shape as the target Mel spectrum. The dimension of the noise matrix is the number of spectrum frames multiplied by the number of Mel bands (80 dimensions). Then, iterative denoising is performed by solving ordinary differential equations. The specific process is as follows: set the total number of iteration steps and calculate the time step of each step. Starting from the initial noise state, in each iteration step, according to the current time step and the aligned text features, input the current state into the diffusion decoder to predict the velocity field direction at that moment. Use the Euler integral method to update the current state according to the predicted velocity field and time step, so that it gradually evolves from noise to clear spectrum. After multiple iterations, a clear Mel spectrum is finally obtained.
[0066] (6) Vocoder conversion stage: The generated Mel spectrogram is input into a neural vocoder (such as HiFi-GAN). The vocoder converts the spectrogram into a time-domain audio waveform signal. The number of audio sampling points output is equal to the number of spectrum frames multiplied by the jump length parameter.
[0067] (7) Post-processing stage (optional): Adjust the volume of the generated audio waveform to ensure that the volume is appropriate; add fade-in and fade-out effects at the beginning and end of the audio to make the sound transition more natural and smooth.
[0068] After the above seven stages of processing, high-quality synthesized speech in Amdo Tibetan dialect is finally obtained. The data flow path of the entire inference process is as follows: the input text is encoded to obtain text features, feature alignment is achieved through duration prediction, then the diffusion model performs multi-step iterative sampling to generate Mel spectrum, and finally the vocoder converts it into an audio waveform output.
[0069] This invention also provides a Tibetan Amdo dialect speech synthesis system, comprising: Model construction was used to obtain a multilingual text-audio pairing dataset; a text-to-speech model was built based on the DIT end-to-end architecture; the text-to-speech model was trained using data from the multilingual text-audio pairing dataset to obtain a multilingual general model.
[0070] The parameter tuning module is used to obtain text-audio pairing data of the Amdo dialect of Tibetan, input the text-audio pairing data of the Amdo dialect of Tibetan into the multilingual general model, and use the full fine-tuning algorithm to update the model parameters of the multilingual general model.
[0071] The speech synthesis module is used to acquire the text of the Amdo dialect of Tibetan to be recognized, and to obtain the synthesized speech of the Amdo dialect of Tibetan by updating the input parameters of the text of the Amdo dialect of Tibetan.
[0072] The modules in the aforementioned Tibetan Amdo dialect speech synthesis system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.
[0073] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a Tibetan Amdo dialect speech synthesis method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.
[0074] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a Tibetan Amdo dialect speech synthesis method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.
[0075] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0079] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.< / unk> < / eos> < / bos> < / pad>
Claims
1. A method for Tibetan Andu dialect speech synthesis, characterized in that, The method comprises the following steps: obtaining a multi-language text-audio paired data set; building a text-to-speech model based on a DIT end-to-end architecture, training the text-to-speech model using data in the multi-language text-audio paired data set, and obtaining a multi-language universal model; obtaining text-audio paired data of the Anduo dialect of Tibetan, inputting the text-audio paired data of the Anduo dialect of Tibetan into the multi-language universal model, and updating model parameters of the multi-language universal model using a full-fledged fine-tuning algorithm; obtaining a Tibetan Anduo dialect text to be recognized, inputting the Tibetan Anduo dialect text to be recognized into the multi-language universal model with updated parameters, and obtaining synthesized speech of the Tibetan Anduo dialect.
2. The Tibetan Ando dialect speech synthesis method according to claim 1, wherein, The text-to-speech model comprises a text encoder, a duration predictor, a diffusion decoder, and a vocoder; when the text-to-speech model is trained using data in the multi-language text-audio paired data set, the text encoder and the duration predictor are trained using a loss function; The diffusion decoder is trained using conditional flow matching (CFM); the total loss is calculated by weighting the losses of the text encoder, the duration predictor, and the diffusion decoder to optimize the pre-trained model and obtain the multi-language universal model.
3. The Tibetan Ando dialect speech synthesis method according to claim 2, wherein, The diffusion decoder is trained using conditional flow matching (CFM), and the real mel-spectrogram is used as the target data in the diffusion decoder; different degrees of Gaussian noise are added to the target data to construct a diffusion path from clear data to pure noise; The model learns the velocity field that recovers from the noise state to the clear spectrum; the optimal transmission path from the noise distribution to the target spectrum distribution is learned based on the text features; and the mel-spectrogram aligned with the input text is gradually generated through a multi-step iterative denoising process based on the optimal transmission path.
4. The method of synthesizing speech of Andu dialect of Tibetan language as claimed in claim 2, wherein, The total loss is calculated by the following formula: ； wherein, is a conditional flow matching loss; is a duration prediction loss; is a pitch prediction loss.
5. The method of synthesizing speech of Andu dialect of Tibetan language as claimed in claim 2, wherein, The model parameters of the multi-language universal model are updated using a full-fledged fine-tuning algorithm, which comprises the following steps: all model parameters of the multi-language universal model obtained in the pre-training stage are used as the initial parameter values for fine-tuning; based on the initial parameter values, a batch of training samples are randomly selected, each batch including text-audio paired data of the Tibetan Anduo dialect; the text is input into the text encoder to obtain high-dimensional feature representation of the text; then the text features are input into the duration predictor to predict the pronunciation duration of each character and calculate the duration prediction loss by comparing with the real duration; the text features are up-sampled and aligned based on the predicted duration to match the number of audio frames; then the diffusion model is trained, which specifically includes: randomly sampling time steps, generating corresponding noise vectors, calculating the interpolation state of noise and target spectrum, inputting the interpolation state into the diffusion decoder to predict the velocity field, calculating the conditional flow matching loss between the predicted velocity field and the real velocity field, and calculating the total loss under Tibetan; and the model parameters of the multi-language universal model are updated based on the total loss under Tibetan.
6. The method of synthesizing speech of a dialect of Zang language according to claim 2, wherein, The Tibetan Anduo dialect text to be recognized is input into the multi-language universal model with updated parameters to obtain synthesized speech of the Tibetan Anduo dialect, which comprises the following steps: The input to-be-recognized Tibetan Amdo dialect text is preprocessed; The preprocessed text is input into the fine-tuned text encoder, which converts the text character sequence into a high-dimensional feature vector representation, obtaining the text features; The text features are input into the fine-tuned duration predictor, which outputs the corresponding pronunciation duration for each text character; According to the predicted duration information, the text features are up-sampled and expanded to match the number of to-be-generated mel-spectrum frames, and the length of the expanded features is equal to the sum of the predicted durations of all characters; A Gaussian noise matrix with the same shape as the target mel-spectrum is initialized, and the iterative denoising is performed by the ordinary differential equation solving method to obtain a clear mel-spectrum graph; The generated mel-spectrum graph is input into the neural vocoder to obtain an audio waveform signal, and a high-quality Tibetan Amdo dialect synthesized speech is obtained.
7. The method as claimed in claim 1, wherein the Tibetan Andu dialect speech synthesis method is characterized by, The method also includes preprocessing the text-audio paired data of the Tibetan Amdo dialect before inputting the text-audio paired data of the Tibetan Amdo dialect into the multi-language universal model, wherein the text-audio paired data includes audio data and text data, and specifically includes the following steps: The audio data is denoised, sentence segmented, trimmed, and resampled; and the text data is uniformly encoded, converted, punctuated, special mark added, and syllable segmented; The mel-spectrum graph is extracted from the preprocessed audio data; and the preprocessed text data is converted into a character sequence processable by the model using a Tibetan character-level or syllable-level tokenizer; The ConvNeXT module is introduced to further process the text features, and the reprocessed text data is aligned with the audio data to obtain training data pairs for fine-tuning.
8. A Tibetan Andu dialect speech synthesis system characterized by, The method includes: Model construction is used to obtain a multi-language text-audio paired dataset; A text-to-speech model is constructed based on the DIT end-to-end architecture; The text-to-speech model is trained using data in the multi-language text-audio paired dataset to obtain a multi-language universal model; A parameter tuning module is used to obtain text-audio paired data of the Tibetan Amdo dialect, input the text-audio paired data of the Tibetan Amdo dialect into the multi-language universal model, and update the model parameters of the multi-language universal model using a full-tuning algorithm; A speech synthesis module is used to obtain to-be-recognized Tibetan Amdo dialect text, input the to-be-recognized Tibetan Amdo dialect text into the multi-language universal model after parameter update, and obtain Tibetan Amdo dialect synthesized speech.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-8. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when loaded by the processor, can execute the steps of the method of any one of claims 1 to 7.