Conba Tibetan speech synthesis front-end modeling method, system, device, medium and program product

Abstract

The invention provides a Kangba Tibetan speech synthesis front-end modeling method, which can be applied to the technical field of artificial intelligence. The front-end modeling method for voice synthesis of the Conba Tibetan language comprises the following steps: acquiring a Conba Tibetan language text and a corresponding real Tibetan language rhythm tag; based on a preset conbar international phonetic symbol transliteration rule, transliteration is conducted on the conbar Tibetan text to determine a conbar Tibetan phoneme vector; based on the three-layer structure modeling rhythm information of the Kangba Tibetan phoneme vector and the hierarchy-sensitive local self-attention mechanism, generating a prediction rhythm tag; and optimizing a rhythm modeling module according to the real Tibetan rhythm tag and the predicted rhythm tag to obtain a target rhythm encoder. The invention also provides a front-end modeling system and equipment for voice synthesis of the conoba Tibetan language, a storage medium and a program product.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to the field of machine learning technology, and more specifically to a front-end modeling method, system, device, medium, and program product for Kham Tibetan speech synthesis. Background Technology

[0002] In the field of Tibetan speech synthesis, the significant differences between Tibetan and Chinese in language structure, phonology, and intonation patterns increase the difficulty of speech synthesis, leading to relatively lagging technological development and numerous challenges. Existing methods have shortcomings in terms of speech naturalness and prosodic expression. For example, waveform splicing speech synthesis technology produces speech with a disjointed feel and poor naturalness; while statistical acoustic models based on Hidden Markov Models (HMMs) improve the fluency of synthesized speech, their ability to model the global context is limited, leaving room for improvement in the quality of synthesized speech. Sequence-to-sequence models based on deep learning improve the quality of synthesized speech, but they do not fully integrate the characteristics of Tibetan, especially lacking adaptability to Kham Tibetan. Therefore, a front-end modeling method for speech synthesis is urgently needed to improve the naturalness and accuracy of synthesized Kham Tibetan speech. Summary of the Invention

[0003] In view of the above problems, this application provides a method, apparatus, equipment, medium and program product for front-end modeling of Kham Tibetan speech synthesis to improve the naturalness and accuracy of Kham Tibetan synthesized speech.

[0004] According to the first aspect of this application, a front-end modeling method for Kham Tibetan speech synthesis is provided, comprising: obtaining Kham Tibetan text and its corresponding real Tibetan prosodic tags;

[0005] The Kham Tibetan text is transcribed based on the preset Kham International Phonetic Alphabet transcription rules to determine the Kham Tibetan phoneme vectors;

[0006] Based on the aforementioned Kham Tibetan phoneme vectors and a hierarchical sensitive local self-attention mechanism, a three-layer structure is used to model prosodic information to generate predicted prosodic labels; and

[0007] The prosodic modeling module is optimized based on the real Tibetan prosodic labels and the predicted prosodic labels to obtain the target prosodic encoder.

[0008] According to an embodiment of this application, the step of transcribing the Kham Tibetan text based on a preset Kham International Phonetic Alphabet transcribing rule to generate a Kham Tibetan phoneme vector includes:

[0009] The Kham Tibetan text is segmented into words, and the syllables of the segmented Tibetan text are divided.

[0010] Based on the preset Kham International Phonetic Alphabet transcription rules, the initials and finals of the divided syllables are converted to generate an International Phonetic Alphabet sequence; and

[0011] The International Phonetic Alphabet sequence is converted into a Kham Tibetan phoneme vector.

[0012] According to embodiments of this application, the three-layer structure modeling of prosodic information based on a hierarchical sensitive local self-attention mechanism to generate predicted prosodic labels includes:

[0013] Local prosodic features are extracted from the input Kham Tibetan phoneme vector through one-dimensional convolution.

[0014] Local attention weights are calculated using dot products to generate prosodic word features;

[0015] Hierarchical multi-head attention is used to capture long-distance dependencies between prosodic words to determine prosodic phrase features;

[0016] The prosodic phrase features are input into the Linformer layer to model the long-distance dependency structure at the intonation phrase level in order to determine the prosodic feature matrix.

[0017] According to an embodiment of this application, the step of optimizing the prosodic modeling module based on the real Tibetan prosodic label and the predicted prosodic label to obtain the target prosodic encoder includes:

[0018] The prosodic feature matrix is ​​fed into a conditional random field layer to calculate the emission matrix and transition matrix;

[0019] The negative log-likelihood loss of the emission matrix and transition matrix is ​​calculated based on the real Tibetan prosodic tags; and

[0020] The parameters of the prosodic modeling module are fine-tuned by minimizing the negative log-likelihood loss to obtain the target prosodic encoder.

[0021] A second aspect of this application provides a method for synthesizing Kham Tibetan speech, the method comprising:

[0022] In response to the Kham Tibetan speech synthesis command, obtain the Kham Tibetan text to be synthesized;

[0023] The Kham Tibetan text to be synthesized is preprocessed to generate a phonetic sequence;

[0024] The phonetic symbol sequence is input into the prosodic encoder to predict the prosodic feature matrix;

[0025] The prosodic feature matrix is ​​subjected to a multilayer perceptron to calculate the scaling factor and translation factor;

[0026] The text embedding features are conditionally normalized using the scaling factor and the translation factor to obtain normalized text embedding features.

[0027] The normalized text embedding features are input into a pre-trained VITS speech synthesis model to output a speech waveform; and

[0028] The speech waveform is output as an audio file, wherein the prosodic encoder is pre-trained according to the method provided in the first aspect of this application.

[0029] A third aspect of this application provides a front-end modeling system for Kham Tibetan speech synthesis, comprising:

[0030] The first acquisition module is used to acquire Kham Tibetan text and its corresponding real Tibetan prosody tags;

[0031] The text transcription module is used to transcribe the Kham Tibetan text based on a preset Kham International Phonetic Alphabet transcription rule in order to determine the Kham Tibetan phoneme vectors;

[0032] The prosodic label prediction module is used to model prosodic information based on the three-layer structure of the Kham Tibetan phoneme vector and the hierarchical sensitive local self-attention mechanism to generate predicted prosodic labels.

[0033] The training module is used to optimize the prosodic modeling module based on the real Tibetan prosodic labels and the predicted prosodic labels to obtain the target prosodic encoder.

[0034] The fourth aspect of this application provides a Kham Tibetan speech synthesis device, comprising:

[0035] The second acquisition module is used to acquire the Kham Tibetan text to be synthesized in response to the Kham Tibetan speech synthesis command;

[0036] The text preprocessing module is used to preprocess the Kham Tibetan text to be synthesized to generate a phonetic sequence;

[0037] The prosodic feature matrix generation module is used to input phonetic symbol sequences into the prosodic encoder to predict the prosodic feature matrix;

[0038] The conditional normalization module is used to calculate the scaling factor and translation factor of the prosodic feature matrix using a multilayer perceptron; and to conditionally normalize the text embedding features using the scaling factor and the translation factor to obtain the normalized text embedding features.

[0039] The first output module is used to embed the normalized text into a pre-trained VITS speech synthesis model with features as input, so as to output a speech waveform; and

[0040] The second output module is used to output the speech waveform as an audio file, wherein the prosodic encoder is pre-trained according to the method provided in the first aspect of this application.

[0041] A fifth aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0042] A sixth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0043] A seventh aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0044] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0045] Figure 1 This illustration schematically depicts an application scenario of the Kham Tibetan speech synthesis front-end modeling method according to an embodiment of this application.

[0046] Figure 2 A flowchart illustrating a front-end modeling method for Kham Tibetan speech synthesis according to an embodiment of this application is shown schematically.

[0047] Figure 3 A flowchart illustrating a method for transcribing Kham Tibetan text according to an embodiment of this application is shown schematically.

[0048] Figure 4 The flowchart illustrating a three-layer structure modeling prosodic information based on a hierarchy-sensitive local self-attention mechanism according to an embodiment of this application is shown in the illustration.

[0049] Figure 5 A flowchart illustrating a method for synthesizing Kham Tibetan speech according to an embodiment of this application is shown schematically.

[0050] Figure 6 A schematic diagram illustrating a framework of a Kham Tibetan speech synthesis model according to an embodiment of this application is shown.

[0051] Figure 7 This schematic diagram illustrates the structural block diagram of a Kham Tibetan speech synthesis front-end modeling system according to an embodiment of this application;

[0052] Figure 8A schematic diagram illustrating the structure of a Kham Tibetan speech synthesis device according to an embodiment of this application is shown.

[0053] as well as

[0054] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a front-end modeling method for Kham Tibetan speech synthesis according to an embodiment of this application. Detailed Implementation

[0055] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0056] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0057] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0058] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0059] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0060] Embodiments of this application provide a front-end modeling method for Kham Tibetan speech synthesis. The method includes: acquiring Kham Tibetan text and its corresponding real Tibetan prosodic tags; transcribing the Kham Tibetan text based on preset Kham International Phonetic Alphabet transcribing rules to determine Kham Tibetan phoneme vectors; modeling prosodic information based on a three-layer structure using a hierarchical sensitive local self-attention mechanism to generate predicted prosodic tags; and optimizing the prosodic modeling module according to the real Tibetan prosodic tags and the predicted prosodic tags to obtain a target prosodic encoder.

[0061] Figure 1 The diagram illustrates an application scenario of the Kham Tibetan speech synthesis front-end modeling method according to an embodiment of this application.

[0062] like Figure 1 As shown, application scenario 100 according to this embodiment may include a speech synthesis application scenario. Network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0063] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0064] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0065] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0066] It should be noted that the Kham Tibetan speech synthesis front-end modeling method provided in this application embodiment can generally be executed by server 105. Correspondingly, the Kham Tibetan speech synthesis front-end modeling system provided in this application embodiment can generally be set up in server 105. The Kham Tibetan speech synthesis front-end modeling method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the Kham Tibetan speech synthesis front-end modeling system provided in this application embodiment can also be set up in a server or server cluster that is different from server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0067] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0068] The following will be based on Figure 1 The described scene, through Figures 2-6 The front-end modeling method for Kham Tibetan speech synthesis according to the embodiments of this application is described in detail.

[0069] Figure 2 A flowchart illustrating a front-end modeling method for Kham Tibetan speech synthesis according to an embodiment of this application is shown. Figure 3 A flowchart illustrating a method for transcribing Kham Tibetan text according to an embodiment of this application is shown.

[0070] like Figure 2 As shown, the Kham Tibetan speech synthesis front-end modeling method of this embodiment includes operations S210 to S240, which can be executed by a server or other computing device.

[0071] In operation S210, obtain the Kham Tibetan text and its corresponding real Tibetan prosody tags.

[0072] In one example, Kham Tibetan text samples can be pre-collected and manually annotated with authentic Tibetan prosodic labels by linguists. These labels are manually added to the corpus by professionals or trained annotators based on the phonetic characteristics and prosodic features of Tibetan. During annotation, the prosodic structure of Tibetan is referenced, dividing the text into prosodic words, prosodic phrases, intonation phrases, etc., and the stress probability of syllables is calculated based on pitch distribution maps, thereby marking the prosodic features of syllables. These manually annotated labels provide the basis for supervised learning of the model, enabling it to learn Tibetan prosodic patterns. During training, the accuracy of prosodic prediction is optimized by calculating the log-likelihood loss of the emission and transition matrices.

[0073] In operation S220, the Kham Tibetan text is transcribed based on the preset Kham International Phonetic Alphabet transcription rules to determine the Kham Tibetan phoneme vectors.

[0074] According to an embodiment of this application, the Kham Tibetan text is segmented into words, and the syllables of the segmented Tibetan text are divided; the initials and finals of the divided syllables are converted based on the preset Kham International Phonetic Alphabet transcription rules to generate an International Phonetic Alphabet sequence; the International Phonetic Alphabet sequence is converted into a Kham Tibetan phoneme vector.

[0075] In one example, such as Figure 3 As shown, using predefined Khampa International Phonetic Alphabet (IPA) transcription rules (as shown in Table 1), each syllable in the text is converted into its corresponding IPA symbol, and these symbols are then mapped to phoneme vectors. Specifically, the open-source TIP-LAS Tibetan automatic word segmentation system can be used to segment the text, and then each word is divided into syllables. According to the predefined transcription rules, the initials and finals of each syllable are converted into corresponding IPA symbols, forming an IPA sequence; each symbol in the IPA sequence is mapped to a unique index, forming a phoneme vector, that is, the phoneme information represented by the symbol is mapped into a numerical vector. Specifically, the IPA Latin character sequence is converted into a phoneme sequence (character encoding); the phoneme sequence is input into the embedding layer to obtain the embedding vector of each phoneme; positional encoding is added to each embedding vector to form the final input representation; the embedding vector with positional information is input into the subsequent layers of the model. In the above operations, because the predefined Khampa IPA transcription rules provide a standardized conversion scheme for Khampa Tibetan text, the ambiguity and errors in transcription are reduced, and the transcription accuracy is improved. This makes the generated International Phonetic Alphabet (IPA) sequence easier for prosodic modeling and speech synthesis modules to process, resulting in speech that is more consistent with the pronunciation characteristics of Kham Tibetan. Simultaneously, the phoneme vectors generated by this rule can be used as training data for the model, improving its learning performance and enhancing its adaptability to the Kham dialect.

[0076] Table 1. Comparison of International Phonetic Alphabet for Kham Tibetan Language

[0077]

[0078] In operation S230, prosodic information is modeled based on the three-layer structure of the Kham Tibetan phoneme vectors and the hierarchical sensitive local self-attention mechanism to generate predicted prosodic labels.

[0079] In one example, phoneme vectors are input into a three-layer structure of Hierarchical-Locality Sensitive Self-Attention (H-LSRA). This H-LSRA captures prosodic information at different levels, modeling prosodic dependencies at the prosodic word, prosodic phrase, and intonation phrase levels to generate predicted prosodic labels. For details, please refer to [link to documentation / reference]. Figure 4 Operations S231 to S234 are shown. Through the three-layer structure of the Hierarchical Sensitive Local Self-Attention Mechanism (H-LSRA) provided in this application, multi-level prosodic information of Kham Tibetan can be effectively captured, improving the detail and accuracy of prosodic modeling.

[0080] In operation S240, the prosodic modeling module is optimized based on the real Tibetan prosodic labels and the predicted prosodic labels to obtain the target prosodic encoder.

[0081] According to an embodiment of this application, the prosodic feature matrix is ​​fed into a conditional random field layer to calculate the emission matrix and the transition matrix; the negative log-likelihood loss of the emission matrix and the transition matrix is ​​calculated based on the real Tibetan prosodic labels; the parameters of the prosodic modeling module are fine-tuned by minimizing the negative log-likelihood loss to obtain the target prosodic encoder.

[0082] In one example, a loss function is calculated using real and predicted prosodic labels. The parameters of the prosodic modeling module are then optimized using backpropagation and gradient descent algorithms to obtain the trained target prosodic encoder. Specifically, the generated prosodic feature matrix is ​​input into a Conditional Random Field (CRF) layer to calculate the emission and transition matrices at each position. Using the real prosodic labels and the calculated emission and transition matrices, a negative log-likelihood loss is calculated. This negative log-likelihood loss is minimized using an optimization algorithm (such as gradient descent), and the parameters of the prosodic modeling module are fine-tuned to obtain the optimized target prosodic encoder.

[0083] After the original Tibetan text sequence is processed through the TinyBERT module, a feature matrix containing Tibetan semantic information and long contextual dependencies is obtained. When this feature matrix enters the CRF module, it is first fed into a linear layer to calculate the weight of the Tibetan prosodic label corresponding to each Tibetan character, which is to calculate the emission matrix.

[0084] in, For input characters, For rhythm tags, and These are the emission matrix and the bias vector, respectively. Represents input characters go through The feature vector output after the module. Then, the emission matrix is ​​used as input to the CRF layer to measure the transition weights between adjacent labels in the label sequence, that is, to calculate the transition matrix:

[0085]

[0086] in, and Representing two adjacent prosody tags, This represents the transition matrix.

[0087] During the fine-tuning phase, the negative log-likelihood loss function is optimized, i.e., minimizing the difference between the actual label sequence and the predicted label sequence. In this application, prosodic modeling involves several adjustable model parameters. The following are some key parameters: H-LSRA related parameters: 1) Number of attention heads: In multi-head attention mechanisms, the number of attention heads affects the model's ability to capture different prosodic features. For example, setting it to 8 attention heads can balance computational complexity and feature extraction richness. 2) Local window size: In locally sensitive self-attention mechanisms, the size of the local window determines the scope of context considered by the model in each computation. Smaller windows can capture more local prosodic features, while larger windows can capture longer-range dependencies. 3) Hidden layer dimension: The dimension of the hidden layer affects the model's expressive power. Higher dimensions can capture more complex prosodic features but also increase computational complexity. CRF (Conditional Random Field) related parameters: 1) Emission matrix parameters: The weights and biases in the emission matrix determine the score of each prosodic feature vector corresponding to different prosodic labels. These parameters are learned through training data, and their initial values ​​can be randomized or based on prior knowledge. 2) Transition matrix parameters: The parameters in the transition matrix represent the probability of transitioning from one prosodic label to another. These parameters are also learned through training data and can be adjusted to optimize the prediction accuracy of the prosodic label sequence.

[0088] By using a three-layer structure of pre-defined Khampa IPA transcription rules and a hierarchy-sensitive local self-attention mechanism (H-LSRA), prosodic information can be accurately extracted from Khampa Tibetan texts to generate predicted prosodic labels. The target prosodic encoder is then obtained by optimizing the prosodic modeling module. This process improves the accuracy of prosodic modeling and the naturalness of the generated speech, ensuring that the synthesized speech better conforms to the linguistic characteristics of Khampa Tibetan.

[0089] Figure 4 The flowchart illustrating a three-layer structure modeling prosodic information based on a hierarchy-sensitive local self-attention mechanism according to an embodiment of this application is shown.

[0090] like Figure 4As shown, operation S230 includes operations S231 to S234.

[0091] In operation S231, the input Kham Tibetan phoneme vector is used to extract local prosodic features through one-dimensional convolution.

[0092] In operation S232, local attention weights are calculated through dot product to generate prosodic word features.

[0093] In one example, the input embedded vector is processed by a one-dimensional convolutional layer to extract local prosodic features, and then local attention weights are calculated by dot product to generate a prosodic word-level prosodic representation, which is then input into a prosodic phrase-level feature extraction module for further processing.

[0094] In operation S233, hierarchical multi-head attention is used to capture long-distance dependencies between prosodic words to determine prosodic phrase features.

[0095] In operation S234, the prosodic phrase features are input into the Linformer layer to model the long-distance dependency structure at the intonation phrase level in order to determine the prosodic feature matrix.

[0096] In one example, due to the long-distance dependencies between prosodic phrases, a hierarchical multi-head attention model is used to model the mapping relationship between prosodic words and prosodic phrases. First, group convolutions are used to aggregate adjacent prosodic words, forming a preliminary representation of the prosodic phrases. Next, multi-head attention is used to compute the relationships between prosodic phrases. During this process, a local window is set for multi-head attention computation, rather than global attention across the entire sentence, which reduces computational cost. Finally, the obtained prosodic phrase feature representations are input into the intonation phrase-level feature extraction module for further final processing. A lightweight Transformer variant, Linformer, is used to model the prosodic features at the intonation phrase level. Linformer reduces the computational complexity of the self-attention mechanism through low-rank approximation, resulting in the final prosodic feature matrix.

[0097] Figure 5 A flowchart illustrating a method for synthesizing Kham Tibetan speech according to an embodiment of this application is shown. Figure 6 A schematic diagram illustrating a framework of a Kham Tibetan speech synthesis model according to an embodiment of this application is provided. Figure 5 As shown, this includes operations S310 to S370.

[0098] When operating S310, in response to the Kham Tibetan speech synthesis command, the Kham Tibetan text to be synthesized is obtained.

[0099] In operation S320, the Kham Tibetan text to be synthesized is preprocessed to generate a phonetic sequence.

[0100] In operation S330, the phonetic symbol sequence is input into the prosodic encoder to predict the prosodic feature matrix.

[0101] According to an embodiment of this application, the prosodic encoder is pre-trained based on the aforementioned Kham Tibetan speech synthesis front-end modeling method.

[0102] In one example, the process of operations S310 to S330 is similar to the aforementioned front-end modeling process: receiving the user's speech synthesis request and obtaining the Kham Tibetan text to be synthesized. The text is then segmented into words, syllables, and converted between initials and finals to generate an International Phonetic Alphabet (IPA) sequence. This generated IPA sequence is then input into a trained prosodic encoder to predict the prosodic feature matrix.

[0103] In operation S340, the prosodic feature matrix is ​​subjected to multilayer perceptron calculation of scaling and translation factors.

[0104] In operation S350, the text embedding features are conditionally normalized using the scaling factor and the translation factor to obtain normalized text embedding features.

[0105] In one example, the text embedding features are obtained based on a text encoder, which is responsible for encoding the input text information into feature representations. It consists of a convolutional neural network and self-attention layers. Convolutional layers can capture local dependencies in character or phoneme sequences, enabling the model to better understand the phonemes of speech. To capture long-range dependencies in the text, a self-attention mechanism is used to help the model focus on more distant contextual information, ensuring that global sentence structure and semantic information are represented when generating speech. Tibetan text is transcribed into a phoneme sequence using Kham Tibetan-IPA and then input into the text encoder for text feature modeling. Simultaneously, the Tibetan text is processed by H-LSRA-CRF to obtain the predicted prosodic feature matrix, X. lsra It will then enter the multilayer perceptron to calculate the scaling factor. Translation factor The formula is as follows:

[0106]

[0107] γ and β adjust the text embedding through conditional normalization, X text For text embedding features, the normalized features Perform subsequent speech synthesis:

[0108]

[0109] In operation S360, the normalized text embedding features are input into the pre-trained VITS speech synthesis model to output a speech waveform.

[0110] In operation S370, the voice waveform is output as an audio file.

[0111] An example illustrates the training process of a Kham Tibetan speech synthesis model. The process involves preparing a training sample set, constructing a dataset containing Kham Tibetan text and its corresponding speech, recorded by professional speakers to ensure accurate pronunciation and natural emotion. Each speech file is matched with its corresponding text transcription and saved in a model-readable format. The text is cleaned, removing irrelevant characters or symbols, and transcribed into phoneme sequences according to predefined rules. Acoustic features, such as Mel-spectrum, are extracted and used as input data for model training. Other relevant features can also be extracted to aid training. At the start of training, the parameters of the VITS model are randomly initialized or initialized using parameters from a pre-trained model. Pre-trained model parameters can be obtained from relevant research or open-source projects. In each training iteration, a batch of data, including text sequences and their corresponding speech features, is loaded from the training sample set. The text sequences are then input into the VITS model, such as... Figure 6 As shown, the model processes the data through components such as a text encoder, prior encoder, posterior encoder, normalized stream, and decoder to generate feature representations of synthesized speech. By introducing a latent variable Z, the model optimizes the prior and posterior encoders, making the generated speech closer to real speech. Simultaneously, based on adversarial training, a discriminator is introduced to distinguish between generated and real speech, further improving the naturalness of the speech. The loss function is calculated by comparing the differences between synthesized and real speech features. The loss function of the VITS model typically includes the mean squared error (MSE) or other similarity measures between generated and real speech features, as well as adversarial loss. These loss function values ​​reflect the model's performance in the current iteration. Based on the loss function value, the gradient is calculated using the backpropagation algorithm, and the model parameters are updated using an optimization algorithm (such as Adam). The optimization algorithm controls the magnitude of parameter updates by adjusting hyperparameters such as the learning rate, gradually reducing the value of the loss function and improving model performance. The model performance is periodically evaluated on the validation set, and the validation loss is calculated. If the validation loss is lower than the training loss, overfitting may occur, requiring adjustment of the regularization parameters or early termination of training. Based on the validation results and the trend of training loss changes, adjust hyperparameters such as learning rate and batch size as needed, or use data augmentation techniques to expand training data to optimize model performance. Stop training when the maximum number of iterations is reached or the validation loss no longer decreases significantly, and save the final model parameters to a file for inference.

[0112] In the reasoning stage, such as Figure 6As shown, the Kham Tibetan text to be synthesized is preprocessed to generate a phonetic symbol sequence, a trained prosodic encoder is used to predict the prosodic feature matrix, and a multilayer perceptron is used to calculate scaling and translation factors to adjust the text embedding features. Finally, a natural and accurate speech waveform is generated and output as an audio file, thus achieving high-quality Kham Tibetan speech synthesis.

[0113] To verify the effectiveness of the transcription scheme and prosodic modeling layer provided in this application, the effects and performance of the Kham Tibetan speech synthesis model provided in this application will be introduced below.

[0114] The Kham Tibetan language dataset was recorded by a professional male Tibetan host. The speaker has received excellent training in hosting, broadcasting, and reading aloud. The speaker is between 30 and 40 years old and is proficient in reading Tibetan books in the Kham dialect.

[0115] The corpus was recorded in a professional broadcast television recording environment. All instruments, equipment, and recording studios used were professionally calibrated before recording. Audio parameters included a signal-to-noise ratio of at least 35dB, a sampling rate of 16kHz, and resampling to 22050Hz with 16-bit precision for subsequent experiments. The data was saved in WAV format. The Kham Tibetan dataset is a single-speaker dataset containing approximately 25 hours of data (9179 sentences) from one male speaker. The ratio of training, testing, and prediction sets is approximately 8:1:1.

[0116] Table 2 shows the word error rate (WER), MCD, and MOS score of the VITS model under different text input conditions:

[0117] Table 2: Comparison of Transcription Schemes and Experimental Results

[0118]

[0119] The experimental results in the table show that KhampaIPA outperforms Wylie transcription on both the Fastspeech2 and VITS models. This indicates that KhampaIPA can represent phoneme features more accurately, thereby reducing the error rate in speech synthesis.

[0120] To verify the effectiveness of the prosodic modeling layer, a series of experiments were designed and compared with those using the VITS model as a baseline. In the training settings, all three models were trained on the same dataset, with the same hyperparameters, and using the same optimizer to ensure fairness. Performance evaluation metrics included fundamental frequency metric F0 RMSE, audio quality metric MOS, and MCD. The results of the prosodic modeling comparison experiments are shown in Table 3.

[0121] Table 3: Comparative Experiment Results of Prosody Modeling

[0122]

[0123] According to the experimental results in Table 3, H-LSRA-VITS performs best in F0 RMSE, MOS, and MCD. This indicates that the H-LSRA-based prosodic encoder can effectively improve the performance of Kham Tibetan speech synthesis.

[0124] Table 4 presents the comparative experimental results of Tibetan speech synthesis models. As can be seen from the results in Table 4, the MOS (Mean Opinion Standard) of speech synthesis based on VITS, FastSpeech, and Tacotron2 is around 3.70, indicating that the transcription scheme and prosodic modeling can achieve good results on current mainstream Tibetan speech synthesis models. Among them, the Tibetan speech synthesis model based on VITS has the best synthesis quality in both MOS value and MCD (Mean Distance).

[0125] Table 4. Comparative Experiment Results of Tibetan Speech Synthesis Models

[0126]

[0127] Based on the aforementioned front-end modeling method for Kham Tibetan speech synthesis, this application also provides a front-end modeling system for Kham Tibetan speech synthesis. The following will combine... Figure 7 The system is described in detail.

[0128] Figure 7 The diagram illustrates the structural block diagram of a front-end modeling system for Kham Tibetan speech synthesis according to an embodiment of this application.

[0129] like Figure 7 As shown, the Kham Tibetan speech synthesis front-end modeling system 700 of this embodiment includes a first acquisition module 710, a text transcription module 720, a prosody tag prediction module 730, and a training module 740.

[0130] The first acquisition module 710 is used to acquire Kham Tibetan text and its corresponding authentic Tibetan prosody tags. In one embodiment, the first acquisition module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0131] The text transcription module 720 is used to transcribe the Kham Tibetan text based on a preset Kham International Phonetic Alphabet transcription rule to determine the Kham Tibetan phoneme vectors. In one embodiment, the text transcription module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0132] The prosodic label prediction module 730 is used to model prosodic information based on the three-layer structure of the Kham Tibetan phoneme vectors and a hierarchical sensitive local self-attention mechanism to generate predicted prosodic labels. In one embodiment, the prosodic label prediction module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0133] The training module 740 is used to optimize the prosodic modeling module based on the real Tibetan prosodic labels and the predicted prosodic labels to obtain the target prosodic encoder. In one embodiment, the training module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0134] According to embodiments of this application, any multiple modules among the first acquisition module 710, text transcription module 720, prosody tag prediction module 730, and training module 740 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the first acquisition module 710, text transcription module 720, prosody tag prediction module 730, and training module 740 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first acquisition module 710, text transcription module 720, prosody tag prediction module 730, and training module 740 may be implemented at least partially as a computer program module that can perform corresponding functions when the computer program module is run.

[0135] Based on the aforementioned method for synthesizing Kham Tibetan speech, this application also provides a device for synthesizing Kham Tibetan speech. The following will be combined with... Figure 8 The device is described in detail.

[0136] Figure 8 A schematic block diagram of a Kham Tibetan speech synthesis device according to an embodiment of this application is shown.

[0137] like Figure 8 As shown, the Kham Tibetan speech synthesis device 800 of this embodiment includes a second acquisition module 810, a text preprocessing module 820, a prosodic feature matrix generation module 830, a conditional normalization module 840, a first output module 850, and a second output module 860.

[0138] The second acquisition module 810 is used to acquire the Kham Tibetan text to be synthesized in response to the Kham Tibetan speech synthesis instruction. In one embodiment, the second acquisition module 810 can be used to perform the operation S310 described above, which will not be repeated here.

[0139] The text preprocessing module 820 is used to preprocess the Kham Tibetan text to be synthesized to generate a phonetic sequence. In one embodiment, the text preprocessing module 820 can be used to perform the operation S320 described above, which will not be repeated here.

[0140] The prosodic feature matrix generation module 830 is used to input the phonetic symbol sequence into the prosodic encoder to predict the prosodic feature matrix. In one embodiment, the prosodic feature matrix generation module 830 can be used to perform the operation S330 described above, which will not be repeated here.

[0141] The conditional normalization module 840 is used to calculate scaling and translation factors on the prosodic feature matrix using a multilayer perceptron; the scaling and translation factors are then used to conditionally normalize the text embedding features to obtain normalized text embedding features. In one embodiment, the conditional normalization module 840 can be used to execute the operations S340 and S350 described above, which will not be repeated here.

[0142] The first output module 850 is used to embed the normalized text into a pre-trained VITS speech synthesis model to output a speech waveform. In one embodiment, the first output module 850 can be used to perform the operation S360 described above, which will not be repeated here.

[0143] The second output module 860 is used to output the speech waveform as an audio file, wherein the prosodic encoder is pre-trained according to the aforementioned method. In one embodiment, the second output module 860 can be used to perform the operation S370 described above, which will not be repeated here.

[0144] According to embodiments of this application, any multiple modules among the second acquisition module 810, text preprocessing module 820, prosodic feature matrix generation module 830, conditional normalization module 840, first output module 850, and second output module 860 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the second acquisition module 810, text preprocessing module 820, prosodic feature matrix generation module 830, conditional normalization module 840, first output module 850, and second output module 860 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the second acquisition module 810, text preprocessing module 820, prosodic feature matrix generation module 830, conditional normalization module 840, first output module 850, and second output module 860 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0145] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a front-end modeling method for Kham Tibetan speech synthesis according to an embodiment of this application.

[0146] like Figure 9 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0147] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0148] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0149] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0150] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0151] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the Kham Tibetan speech synthesis front-end modeling method provided in the embodiments of this application.

[0152] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0153] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0154] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0155] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A front-end modeling method for Kham Tibetan speech synthesis, characterized in that, The method includes: Obtain Kham Tibetan text and its corresponding authentic Tibetan prosodic tags; The Kham Tibetan text is transcribed based on the preset Kham International Phonetic Alphabet transcription rules to determine the Kham Tibetan phoneme vectors; Based on the aforementioned Kham Tibetan phoneme vectors and a hierarchical sensitive local self-attention mechanism, a three-layer structure is used to model prosodic information to generate predicted prosodic labels; and The prosodic modeling module is optimized based on the real Tibetan prosodic labels and the predicted prosodic labels to obtain the target prosodic encoder.

2. The method according to claim 1, characterized in that, The process of transcribing the Kham Tibetan text based on preset Kham International Phonetic Alphabet transcribing rules to generate Kham Tibetan phoneme vectors includes: The Kham Tibetan text is segmented into words, and the syllables of the segmented Tibetan text are divided. Based on the preset Kham International Phonetic Alphabet transcription rules, the initials and finals of the divided syllables are converted to generate an International Phonetic Alphabet sequence; and The International Phonetic Alphabet sequence is converted into a Kham Tibetan phoneme vector.

3. The method according to claim 1, characterized in that, The three-layer structure modeling of prosodic information based on the Kham Tibetan phoneme vectors and a hierarchical sensitive local self-attention mechanism to generate predicted prosodic labels includes: Local prosodic features are extracted from the input Kham Tibetan phoneme vector through one-dimensional convolution. Local attention weights are calculated using dot products to generate prosodic word features; Hierarchical multi-head attention is used to capture long-distance dependencies between prosodic words to determine prosodic phrase features; and The prosodic phrase features are input into the Linformer layer to model the long-distance dependency structure at the intonation phrase level in order to determine the prosodic feature matrix.

4. The method according to any one of claims 1 to 3, characterized in that, The optimization of the prosodic modeling module based on the real Tibetan prosodic labels and the predicted prosodic labels to obtain the target prosodic encoder includes: The prosodic feature matrix is ​​fed into a conditional random field layer to calculate the emission matrix and transition matrix; The negative log-likelihood loss of the emission matrix and transition matrix is ​​calculated based on the real Tibetan prosodic tags; and The parameters of the prosodic modeling module are fine-tuned by minimizing the negative log-likelihood loss to obtain the target prosodic encoder.

5. A method for synthesizing Kham Tibetan speech, characterized in that, The method includes: In response to the Kham Tibetan speech synthesis command, obtain the Kham Tibetan text to be synthesized; The Kham Tibetan text to be synthesized is preprocessed to generate a phonetic sequence; The phonetic symbol sequence is input into the prosodic encoder to predict the prosodic feature matrix; The prosodic feature matrix is ​​subjected to a multilayer perceptron to calculate the scaling factor and translation factor; The text embedding features are conditionally normalized using the scaling factor and the translation factor to obtain normalized text embedding features. The normalized text embedding features are input into a pre-trained VITS speech synthesis model to output a speech waveform; and The speech waveform is output as an audio file, wherein the prosodic encoder is pre-trained using the method according to any one of claims 1 to 4.

6. A front-end modeling system for Kham Tibetan speech synthesis, characterized in that, The system includes: The first acquisition module is used to acquire Kham Tibetan text and its corresponding real Tibetan prosody tags; The text transcription module is used to transcribe the Kham Tibetan text based on a preset Kham International Phonetic Alphabet transcription rule in order to determine the Kham Tibetan phoneme vectors; The prosodic label prediction module is used to model prosodic information based on the three-layer structure of the Kham Tibetan phoneme vector and the hierarchical sensitive local self-attention mechanism to generate predicted prosodic labels. The training module is used to optimize the prosodic modeling module based on the real Tibetan prosodic labels and the predicted prosodic labels to obtain the target prosodic encoder.

7. A Kham Tibetan speech synthesis device, characterized in that, The device includes: The second acquisition module is used to acquire the Kham Tibetan text to be synthesized in response to the Kham Tibetan speech synthesis command; The text preprocessing module is used to preprocess the Kham Tibetan text to be synthesized to generate a phonetic sequence; The prosodic feature matrix generation module is used to input phonetic symbol sequences into the prosodic encoder to predict the prosodic feature matrix; The conditional normalization module is used to calculate the scaling factor and translation factor of the prosodic feature matrix using a multilayer perceptron; and to conditionally normalize the text embedding features using the scaling factor and the translation factor to obtain the normalized text embedding features. The first output module is used to embed the normalized text into a pre-trained VITS speech synthesis model with features as input, so as to output a speech waveform; and The second output module is used to output the speech waveform as an audio file, wherein the prosodic encoder is pre-trained by the method according to any one of claims 1 to 4.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 5.

Images