A method for generating parallel corpus datasets of the three major Tibetan dialects based on a multi-dialect text-to-speech model

Abstract

This invention proposes a method for generating parallel corpus datasets of the three major Tibetan dialects based on a multi-dialect text-to-speech model, belonging to the field of Tibetan language processing. This method innovatively embeds dialect IDs into a conditional variational autoencoder framework and dynamically replaces the feedforward network in the Transformer through a designed Switchffn module, achieving adaptive switching of dialect features. Simultaneously, it integrates an improved BigVGAN vocoder, enabling the generated speech to possess the acoustic features of specific dialects. Compared to traditional methods, this invention can achieve high-quality speech generation of the three major dialects in a single model, without relying on strictly aligned parallel corpus data, significantly reducing the development cost and data annotation cost of multi-dialect speech synthesis systems. This method effectively avoids overfitting problems in low-resource scenarios, and the generated parallel corpus outperforms existing technologies in terms of naturalness and dialect feature preservation, providing an effective technical solution for Tibetan multi-dialect speech synthesis and language resource protection.

Description

Technical Field

[0001] This invention belongs to the field of Tibetan language processing, specifically involving a method for generating parallel corpus datasets of the three major Tibetan dialects based on a multi-dialect text-to-speech model. Background Art

[0002] The research and development of Tibetan multi-dialect speech generation technology is rooted in the intersection of linguistics and artificial intelligence, aiming to address the complex challenges posed by the diversity of Tibetan dialects. Traditional speech synthesis techniques, such as parametric synthesis and concatenation synthesis, were once the foundation of this field. However, these methods rely on manually annotated phonetic features and fixed prosodic rules, making it difficult to adapt to the significant differences between the three major Tibetan dialects (Ü-Tsang, Kham, and Amdo). For example, the tonal system of Ü-Tsang contrasts sharply with the toneless nature of Amdo, while Kham contains sub-dialectal phonological differentiation. With breakthroughs in deep learning technology, end-to-end models such as Tacotron and WaveNet have been gradually introduced. However, these models need to be modified to suit the characteristics of Tibetan, such as embedding pitch contour modeling modules in tonal dialects or strengthening duration and stress prediction mechanisms in toneless dialects.

[0003] Data scarcity is another core challenge, especially the lack of low-resource corpora such as sub-dialects within Kham, prompting researchers to explore transfer learning and data augmentation strategies. Currently, a common approach is to first pre-train the model using the Ü-Tsang dialect, and then fine-tune the model using a small amount of Kham and Amdo dialect data.

[0004] At the model architecture level, cross-dialect speech generation often employs a multi-task learning framework, sharing underlying acoustic features and utilizing branch networks to capture pronunciation details specific to each dialect. For example, to address the laryngomalacia phenomenon in the Amdo dialect, a pronunciation rule engine needs to be embedded in the front-end text analysis to ensure the accuracy of pronunciation feature conversion; while for the tonal variations in the Ü-Tsang dialect, a hierarchical prosodic prediction network needs to be designed to simulate intonation fluctuations. The introduction of unsupervised learning techniques further reduces the reliance on dialect-annotated data. Extracting dialect-invariant features from speech using self-supervised models (such as wav2vec 2.0) and then adapting them with a small amount of annotated data has become an emerging research direction.

[0005] Current technologies still face profound challenges in modeling cultural contexts. For example, the unique expressions of religious terminology or poetic rhythm are difficult to quantify using existing acoustic parameters. Cutting-edge research is attempting to integrate multimodal information, generating more culturally adaptable dialect variants by combining textual semantics with speaker identity features, or utilizing latent variable control to achieve multi-dialectal style transfer within the same text. Breakthroughs in these technologies rely not only on algorithmic optimization but also on linguists providing fine-grained phonological rules and pragmatic knowledge. This will improve the naturalness of the synthesized speech while preventing the loss of dialectal uniqueness due to excessive "Tibetanization." In the future, with the deepening of low-resource learning methods and interdisciplinary collaboration, multi-dialect Tibetan speech generation is expected to achieve wider applications in education, cultural heritage protection, and other fields. Summary of the Invention

[0006] The purpose of this invention is to provide a method for generating parallel corpus datasets of the three major Tibetan dialects based on a multi-dialect text-to-speech model, which solves the bottlenecks of traditional methods in terms of pronunciation diversity, prosodic modeling and low resource adaptation, and significantly reduces the development cost of multi-dialect speech synthesis systems and the annotation cost of parallel corpus datasets.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A method for generating parallel corpus datasets of the three major Tibetan dialects based on a multi-dialect text-to-speech model includes the following steps:

[0009] Step 1: Input Tibetan dialect ID, reference audio and Tibetan text, and fuse them using a text encoder to obtain text hidden features. The text hidden features are then segmented into the natural logarithm of the mean and variance of the prior hidden feature distribution.

[0010] Step 2: Obtain hidden features from the Mel spectrogram of the reference audio using a posterior encoder;

[0011] Step 3: Map the hidden features to a Gaussian distribution space to obtain the posterior hidden features; perform hard alignment estimation on the natural logarithms of the mean and variance of the posterior and prior hidden feature distributions to obtain the alignment matrix;

[0012] Step 4: Input the hidden text features into the duration predictor to obtain the predicted duration, and input them into the duration discriminator along with the alignment matrix. Through adversarial generation training, the prediction of sound duration is achieved.

[0013] Step 5: Fuse the Tibetan dialect ID, reference audio, and hidden features to obtain the fused features. Input the fused features into the vocoder and output the generated audio.

[0014] Step 6: Optimize the training using the loss function to obtain a trained multi-dialect text-to-speech model, and use this model to synthesize parallel corpus datasets of the three major Tibetan dialects.

[0015] Furthermore, step 1 is detailed as follows:

[0016] Step 101: Encode the Tibetan dialect ID, reference audio, and Tibetan text into feature vectors of the same dimension;

[0017] Tibetan text is mapped to text features through an embedding layer, then context-dependent text features are extracted by a text encoder, and upsampling is performed to obtain the prior feature distribution. ;

[0018] Tibetan dialect IDs achieve dialect-aware text encoding through a learnable dialect tag embedding matrix, thereby obtaining dialect features;

[0019] The reference audio is encoded by an encoder of a convolutional neural network to obtain voiceprint features;

[0020] Step 102: Add the text features, voiceprint features, and dialect features to obtain the same-dimensional fused features;

[0021] Step 103: Transpose the features obtained in step S102 and pass them through a multi-layer self-attention mechanism to obtain the hidden text features. Then text hiding features After the convolutional layer, it is segmented into the mean of the prior hidden feature distribution. The natural logarithm of variance feature.

[0022] Furthermore, the vocoder is specifically a BigVGAN vocoder, and its workflow is as follows:

[0023] Use a one-dimensional convolutional layer to fuse features The system expands the dimensions by inputting multiple parallel stacked Snake modules, processing them, and then summing them. It then activates the Snake1d activation function, adjusts the dimensions using a one-dimensional convolutional layer, and outputs the generated audio after passing through the Tanh activation function.

[0024] Furthermore, the hard alignment estimation uses a monotonic alignment search algorithm.

[0025] Furthermore, the self-attention mechanism includes a self-attention layer and a feedforward network structure. The feedforward network is split into a general feedforward network and a dialect feedforward network. The general feedforward network is used to model the commonalities of Tibetan dialects, while the dialect feedforward network is used to model the features of specific dialects. The processing results are then summed and integrated, specifically including the following steps:

[0026] Step 1031: Input the output features of the self-attention layer into the multi-head self-attention mechanism to obtain the first feature;

[0027] Step 1032: Input the first feature into a general feedforward network consisting of a single linear layer to obtain the output common hidden feature;

[0028] Step 1033: Select a linear layer based on the dialect ID, input the first feature, and output the dialect ID feature; specifically, there are 3 linear layers, each corresponding to a dialect ID;

[0029] Step 1034: Add the common hidden features and the dialect ID features to obtain the fused hidden features output by the feedforward network. The fused hidden features output by the self-attention mechanism of the final layer are used as the text hidden features. .

[0030] Furthermore, the input of the Snake module specifically undergoes one-dimensional transposed convolution, upsampling, Snake1d activation function, one-dimensional downsampling, and one-dimensional dilated convolution.

[0031] Furthermore, the loss function is specifically as follows:

[0032] ;

[0033] in, For the total loss, The reconstruction loss is obtained by calculating the L1 distance between the Mel spectrograms of the tagged audio and the generated audio; The reconstruction loss is the sum of the posterior and prior distributions. For the loss of the duration predictor, Let G be the least squares loss of the vocoder, where G is the generator of the vocoder; This represents the feature matching loss of the vocoder.

[0034] Furthermore, the reconstruction loss between the posterior distribution and the prior distribution The calculation formula is expressed as follows:

[0035] ;

[0036] Where z is the hidden feature. Mel spectrogram of the labeled audio. Here, A is the alignment matrix, and A is the text condition. Represents the posterior distribution of the hidden features. Represents the prior distribution of the hidden features;

[0037] The loss of the duration predictor The calculation formula is expressed as follows:

[0038] ;

[0039] Where d represents the duration of the pronunciation feature in the acoustic feature sequence. These represent the alignment boundaries, corresponding to the start and end frame positions of the pronunciation features in time, respectively. : Indicates the duration of a pronunciation feature. For the prior duration distribution, For the posterior boundary distribution, For the posterior distribution The following expectations;

[0040] Least squares loss The calculation formula is expressed as follows:

[0041] ;

[0042] in For vocoder generator, For the discriminator of the vocoder, This represents the average value of the loss across different audio frequencies;

[0043] Feature matching loss The calculation formula is expressed as follows:

[0044] ;

[0045] Where T is the total number of layers in the discriminator. For output with The first feature number discriminator Feature map of the layer The average value of the matching loss for different audio features.

[0046] Furthermore, the specific steps for synthesizing parallel corpus datasets of the three major Tibetan dialects using this model include:

[0047] Step 601: The text and dialect ID are encoded separately through an embedding layer, while the reference audio is processed by a convolutional encoder to extract voiceprint features. The three are then fused and encoded by a text encoder to obtain the text hidden features. Upsampling is then performed to obtain the mean of the feature distribution. The natural logarithm of variance Sampling is performed on it;

[0048] Step 602: Predict the duration based on the hidden features of the input text using the duration predictor. The time series features are obtained by upsampling the sampling results. The hidden feature z is obtained by changing its feature distribution through the inverse process of normalization flow;

[0049] Step 603: Hide features The input is given to the vocoder, and the output is the generated speech.

[0050] Furthermore, the workflow of the trained multi-dialect text-to-speech model is as follows:

[0051] Step S601: Input Tibetan text and reference audio;

[0052] Step S602: Initialize dialect id = 0;

[0053] Step S603: Input the Tibetan text, reference audio, and dialect ID into the Tibetan text-to-speech model;

[0054] Step S604: Output the generated audio and the corresponding dialect ID;

[0055] Step S605: Determine if id is less than 3. If yes, proceed to S606; otherwise, proceed to S607.

[0056] Step S606: id = id + 1;

[0057] Step S607: End.

[0058] This invention employs a deep learning approach. Based on a conditional variational autoencoder model, this method incorporates dialect information to generate multi-dialect speech. A switchffn module is built to replace the feedforward network module in the transformer block, enabling dialect switching within the model. Drawing inspiration from the BigVGAN vocoder model, Tibetan dialect IDs are fused with the BigVGAN vocoder to generate speech with dialectal characteristics.

[0059] The advantage of this algorithm is that it can generate speech in three dialects in an end-to-end model without relying on speech data aligned with the corpus, and can utilize the corpus data of the three dialects, thus solving the problem of model overfitting under low-resource corpus conditions. Attached Figure Description

[0060] To more clearly illustrate the technical solution of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. Wherein:

[0061] Figure 1 This is a training diagram of the Tibetan multi-dialect text-to-speech model of the present invention;

[0062] Figure 2 This is a schematic diagram illustrating the reasoning process of the Tibetan multi-dialect text-to-speech model of the present invention;

[0063] Figure 3 This is a schematic diagram of the dialect ID fusion module of the present invention;

[0064] Figure 4 This is a schematic diagram of the switchffn module of the present invention;

[0065] Figure 5 This is a schematic diagram of the BigVGAN vocoder that integrates dialect IDs according to the present invention;

[0066] Figure 6 This is a flowchart of the Tibetan multi-dialect text-to-speech model for synthesizing parallel corpus datasets of the three major Tibetan dialects, as described in this invention. Detailed Implementation

[0067] To more clearly illustrate the technical solution of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and specific examples. The specific embodiments of the present invention are not limited to the following examples, and appropriate changes and modifications can be made according to actual needs without affecting the substance and scope of the present invention.

[0068] A method for generating parallel corpus datasets of the three major Tibetan dialects based on a multi-dialect text-to-speech model includes the following steps:

[0069] The first step is the training process for a multi-dialect text-to-speech model:

[0070] Step 1: Input Tibetan dialect ID, reference audio and Tibetan text, fuse them using a text encoder to obtain text hidden features, and segment the text hidden features into the natural logarithm of the mean and variance of the prior hidden feature distribution;

[0071] Step 101: Encode the Tibetan dialect ID, reference audio, and Tibetan text into feature vectors of the same dimension; specifically, the Tibetan text is mapped to an embedding layer. A dense vector of dimension 192 is used as the text feature, where `batch` is the batch size, `length` is the sentence length, and the feature length of a single Tibetan character is 192. Subsequently, a text encoder extracts context-dependent text representations, which are then upsampled to obtain the prior feature distribution. Take samples from it;

[0072] Tibetan dialect IDs are mapped to the following through the embedding layer: The dialect features are obtained by using a learnable dialect label embedding matrix (Ü-Tsang=0, Kham=1, Amdo=2) to achieve dialect-aware text encoding.

[0073] The reference audio is obtained after being encoded by an encoder of a convolutional neural network. Dimensional voiceprint features;

[0074] Step 102: Add the text features, voiceprint features, and dialect features to obtain... Dimensional fusion features;

[0075] Step 103: Transpose the features to The hidden text features are obtained by passing the dimensional vector through multiple layers of self-attention mechanism. At the same time, hide text features After increasing the dimensionality using a one-dimensional convolutional layer, the data is segmented into the mean of the prior hidden feature distribution. The natural logarithm of variance feature.

[0076] like Figure 3 As shown, the self-attention mechanism in the text encoder can encode text features more flexibly through self-attention layers and feedforward network structures, but it lacks the ability to model specific dialects. Therefore, a switchffn structure is constructed to replace the feedforward network structure in the self-attention mechanism, enabling the model to dynamically switch the corresponding module based on the dialect ID. Specifically, the feedforward network is split into a general feedforward network and a dialect feedforward network. The general feedforward network is used to model the commonalities of Tibetan dialects, while the dialect feedforward network is used to model the features of specific dialects. The processing results are then summed and integrated. The dialect ID is used to dynamically switch the dialect feedforward network, allowing the model to adapt to the speech generation of the corresponding dialect while ensuring that the model's inference speed remains unchanged. Figure 4 As shown, the specific steps for handling multi-dialect switchffn modules include:

[0077] Step 1031: Convert the output features of the self-attention layer The first feature is obtained by inputting a multi-head self-attention mechanism. ;

[0078] Step 1032: [Illegible text - likely related to a feature] The input is a feedforward network consisting of a single linear layer, and the output is a common hidden feature. ;

[0079] Step 1033: Select a linear layer based on the dialect ID, and apply the first feature... Input and output dialect ID features Specifically, there are 3 linear layers, each corresponding to a dialect ID;

[0080] Step 1034: Share hidden features Dialect ID features The summation yields the module output fused hidden features, and the fused hidden features output by the final layer's self-attention mechanism serve as the final text hidden features. .

[0081] Step 2: The Mel spectrogram of the input reference audio is used to obtain the mean and variance of the hidden features of the speech from the posterior encoder, and then sampling is performed to obtain the hidden features. ;

[0082] Step 3: Use the Normalized Flow module to extract hidden features from complex distributions. Mapping to a Gaussian distribution space yields the posterior hidden features. Assuming that both prior and posterior features conform to a Gaussian distribution, the Monotonic Alignment Search (MAS) algorithm is used to extract the posterior hidden features from the Gaussian distribution space. Distribution after the prior text encoder output and Hard alignment estimation is performed on the features to obtain the alignment matrix. This indicates the duration of each phonetic feature;

[0083] Step 4: Use a duration predictor to hide features from the input text Predicted duration , and align it with the alignment matrix In the input duration discriminator, the prediction of sound duration is achieved through adversarial generative training;

[0084] Step 5: Combine Tibetan dialect ID, reference audio, and hidden features By performing fusion, fusion characteristics are obtained. This allows the generated speech to possess the characteristics of the corresponding dialect; the fusion features Instead of inputting the spectrogram into the vocoder, change the one-dimensional convolution dimension in the vocoder to match the hidden features. The system adapts to the generated audio output. Figure 5 This is a schematic diagram of the bigvgan vocoder that integrates dialect IDs according to the present invention. As shown in the figure, the process includes the following steps;

[0085] Step 501: Use a one-dimensional convolutional layer to fuse features Expand the dimensions;

[0086] Step 502: The features are input into multiple parallel stacked Snake modules, processed and then added together. The input of the Snake module specifically goes through one-dimensional transposed convolution, upsampling, Snake1d activation function, one-dimensional downsampling, and one-dimensional dilated convolution.

[0087] Step 503: Activate the audio using the Snake1d activation function, adjust the dimensions using a one-dimensional convolutional layer, map the values ​​to the range [-1, 1] using the Tanh activation function, and output the generated audio.

[0088] Step 6: Optimize the model using the loss function;

[0089] Calculate the Mel spectrograms of the tagged audio and the generated audio respectively. and ,Compare and The reconstruction loss of the model is obtained from the L1 distance between them. :

[0090] (1)

[0091] in Represents the L1 distance, used to compare hidden features. The difference between the model and its prior distribution is the model's reconstruction loss. :

[0092] (2)

[0093] in, Mel spectrogram of the labeled audio. For textual conditions, A is the alignment matrix obtained by the MAS alignment algorithm. Represents the posterior distribution of the hidden features. Represents the prior distribution of the hidden features;

[0094] Based on the variational autoencoder model, the loss of the duration predictor is obtained. :

[0095] (3)

[0096] Where d represents the duration of the pronunciation feature in the acoustic feature sequence. These represent the alignment boundaries, corresponding to the start and end frame positions of the pronunciation features in time, respectively. : Indicates the duration of a pronunciation feature. For the prior duration distribution, For the posterior boundary distribution, For the posterior distribution The expectations below.

[0097] Calculate the least squares loss of the vocoder discriminator :

[0098] (4)

[0099] in For vocoder generator, For the discriminator of the vocoder, For tagged audio, For predicting audio.

[0100] Calculate the least squares loss of the vocoder :

[0101] (5)

[0102] in This represents the average value of the loss for different audio frequencies.

[0103] Calculate the feature matching loss of the vocoder :

[0104] (6)

[0105] Where T is the total number of layers in the discriminator. For output with The first feature number discriminator Feature map of the layer This represents the average loss for matching different audio features;

[0106] The total loss for model training is obtained by summing the individual losses. :

[0107] (7)

[0108] By training and optimizing the model, a well-trained multi-dialect text-to-speech model was obtained. This model was then used to synthesize parallel corpus datasets of the three major Tibetan dialects (Ü-Tsang, Amdo, and Kham). The specific reasoning process is as follows: Figure 2 As shown, the process includes the following steps:

[0109] S201: The text and dialect ID are encoded separately through an embedding layer, while the reference audio is processed by a convolutional encoder to extract voiceprint features. These three are then fused to obtain a fused feature. Subsequently, the fused feature is encoded using a text encoder, and upsampling is performed to obtain the mean of the feature distribution. The natural logarithm of variance Sampling is performed on it;

[0110] S202: Predict duration using a duration predictor based on latent features of the input text. The time series features are obtained by upsampling the sampling results. By changing its characteristic distribution through the inverse process of normalized flow, we obtain... ;

[0111] S203: Will The input to the BigVGAN vocoder also incorporates dialect IDs and reference audio features, and the output is the generated speech.

[0112] Figure 6 The flowchart of the Tibetan multi-dialect text-to-speech model for synthesizing parallel corpus datasets of the three major Tibetan dialects is shown in the figure. The process includes the following steps.

[0113] S601: Input Tibetan text and reference audio;

[0114] S602: Initialize dialect id = 0;

[0115] S603: Input the text, reference audio, and dialect ID into the Tibetan text-to-speech model;

[0116] S604: Output the generated audio and the corresponding dialect ID;

[0117] S605: Determine if id is less than 3. If yes, proceed to S606; otherwise, proceed to S607.

[0118] S606: id = id + 1;

[0119] S607: End.

[0120] The specific embodiments of the present invention have been described above, but are merely illustrative examples, and the related interface diagrams are schematic diagrams. Various modifications can be made to these embodiments in practical applications without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

Claims

1. A method for generating a parallel corpus dataset of the three major Tibetan dialects based on a multi-dialect text-to-speech model, characterized in that, Includes the following steps: Step 1: Input Tibetan dialect ID, reference audio and Tibetan text, fuse them using a text encoder to obtain text hidden features, and segment the text hidden features into the natural logarithm of the mean and variance of the prior hidden feature distribution; Step 2: Obtain hidden features from the Mel spectrogram of the reference audio using a posterior encoder; Step 3: Map the hidden features to a Gaussian distribution space to obtain the posterior hidden features; The alignment matrix is ​​obtained by hard-aligning the natural logarithms of the mean and variance of the posterior and prior hidden feature distributions. Step 4: Input the hidden text features into the duration predictor to obtain the predicted duration, and input them into the duration discriminator along with the alignment matrix. Through adversarial generation training, the prediction of sound duration is achieved. Step 5: Fuse the Tibetan dialect ID, reference audio, and hidden features to obtain the fused features. Input the fused features into the vocoder and output the generated audio. Step 6: Optimize the training using the loss function to obtain a well-trained multi-dialect text-to-speech model, and use this model to synthesize parallel corpus datasets of the three major Tibetan dialects; The loss function is specifically as follows: ； in, is the total loss, The reconstruction loss is obtained by calculating the L1 distance between the Mel spectrograms of the tagged audio and the generated audio; The reconstruction loss is the sum of the posterior and prior distributions. For the loss of the duration predictor, Let G be the least squares loss of the vocoder, where G is the generator of the vocoder; For the feature matching loss of the vocoder; The reconstruction loss between the posterior and prior distributions The calculation formula is expressed as follows: ； Where z is the hidden feature. Mel spectrogram of the labeled audio. Here, A is the alignment matrix, and A is the text condition. Represents the posterior distribution of the hidden features. Represents the prior distribution of the hidden features; The loss of the duration predictor The calculation formula is expressed as follows: ； Where d represents the duration of the pronunciation feature in the acoustic feature sequence. These represent the alignment boundaries, corresponding to the start and end frame positions of the pronunciation features in time, respectively. : Indicates the duration of a pronunciation feature. For the prior duration distribution, For the posterior boundary distribution, For the posterior distribution The following expectations; Least squares loss The calculation formula is expressed as follows: ； in For vocoder generator, For the discriminator of the vocoder, This represents the average value of the loss across different audio frequencies; Feature matching loss The calculation formula is expressed as follows: ； Where T is the total number of layers in the discriminator. For output with The first feature number discriminator Feature map of the layer This represents the average value of the matching loss for different audio features. Let z be the labeled audio, and z be the hidden feature.

2. The method for generating a parallel corpus dataset of the three major Tibetan dialects based on a multi-dialect text-to-speech model according to claim 1, characterized in that, Step 1 is described in detail as follows: Step 101: Encode the Tibetan dialect ID, reference audio, and Tibetan text into feature vectors of the same dimension; Tibetan text is mapped to text features through an embedding layer, then context-dependent text features are extracted by a text encoder, and upsampling is performed to obtain the prior feature distribution. ; Tibetan dialect IDs achieve dialect-aware text encoding through a learnable dialect tag embedding matrix, thereby obtaining dialect features; The reference audio is encoded by an encoder of a convolutional neural network to obtain voiceprint features; Step 102: Add the text features, voiceprint features, and dialect features to obtain the same-dimensional fused features; Step 103: Transpose the features obtained in Step 102 and pass them through a multi-layer self-attention mechanism to obtain the hidden text features. Then text hiding features After the convolutional layer, it is segmented into the mean of the prior hidden feature distribution. The natural logarithm of variance feature.

3. The method for generating a parallel corpus dataset of the three major Tibetan dialects based on a multi-dialect text-to-speech model according to claim 2, characterized in that, The vocoder is specifically a BigVGAN vocoder, and its workflow is as follows: Use a one-dimensional convolutional layer to fuse features To expand the dimension, multiple parallel stacked Snake modules are input, processed, and then added together; Then, the Snake1d activation function is used for activation, the dimensions are adjusted using a one-dimensional convolutional layer, and after passing through the Tanh activation function, the generated audio is output.

4. The method for generating a parallel corpus dataset of the three major Tibetan dialects based on a multi-dialect text-to-speech model according to claim 3, characterized in that, The hard alignment estimation uses a monotonic alignment search algorithm.

5. The method for generating a parallel corpus dataset of the three major Tibetan dialects based on a multi-dialect text-to-speech model according to claim 4, characterized in that, The self-attention mechanism includes a self-attention layer and a feedforward network structure. The feedforward network is divided into a general feedforward network and a dialect feedforward network. The general feedforward network is used to model the commonalities of Tibetan dialects, while the dialect feedforward network is used to model the features of specific dialects. The processing results are then summed and integrated, specifically including the following steps: Step 1031: Input the output features of the self-attention layer into the multi-head self-attention mechanism to obtain the first feature; Step 1032: Input the first feature into a general feedforward network consisting of a single linear layer to obtain the output common hidden feature; Step 1033: Select a linear layer based on the dialect ID, input the first feature, and output the dialect ID feature; specifically, there are 3 linear layers, each corresponding to a dialect ID; Step 1034: Add the common hidden features and the dialect ID features to obtain the fused hidden features output by the feedforward network. The fused hidden features output by the self-attention mechanism of the final layer are used as the text hidden features. .

6. The method for generating a parallel corpus dataset of the three major Tibetan dialects based on a multi-dialect text-to-speech model according to claim 5, characterized in that, The input to the Snake module specifically undergoes one-dimensional transposed convolution, upsampling, Snake1d activation function, one-dimensional downsampling, and one-dimensional dilated convolution.

7. The method for generating a parallel corpus dataset of the three major Tibetan dialects based on a multi-dialect text-to-speech model according to claim 6, characterized in that, The specific steps involved in synthesizing parallel corpus datasets of the three major Tibetan dialects using this model are as follows: Step 601: The text and dialect ID are encoded separately through an embedding layer, while the reference audio is processed by a convolutional encoder to extract voiceprint features. The three are then fused and encoded by a text encoder to obtain the hidden text features. Upsampling is then performed to obtain the mean of the feature distribution. The natural logarithm of variance Sampling is performed on it; Step 602: Predict the duration based on the hidden features of the input text using the duration predictor. The time series features are obtained by upsampling the sampling results. The hidden feature z is obtained by changing its feature distribution through the inverse process of normalization flow; Step 603: Hide features The input is given to the vocoder, and the output is the generated speech.

8. The method for generating a parallel corpus dataset of the three major Tibetan dialects based on a multi-dialect text-to-speech model according to claim 7, characterized in that, The workflow of the trained multi-dialect text-to-speech model is as follows: Step S601: Input Tibetan text and reference audio; Step S602: Initialize dialect id = 0; Step S603: Input the Tibetan text, reference audio, and dialect ID into the Tibetan text-to-speech model; Step S604: Output the generated audio and the corresponding dialect ID; Step S605: Determine if id is less than 3. If yes, proceed to S606; otherwise, proceed to S607. Step S606: id = id + 1; Step S607: End.

Images