A method and device for Mongolian speech synthesis based on character component modeling

Abstract

This invention provides a method and apparatus for Manchu speech synthesis based on character component modeling, belonging to the field of speech synthesis technology. The method includes: converting Manchu text into Latin characters according to transcription rules, generating a character embedding sequence, and generating feature templates containing dimensions such as fundamental frequency and pitch band based on the acoustic characteristics of Manchu pronunciation; concatenating the two templates into a high-dimensional input tensor, which is then fed into a pre-trained duration prediction network; constructing a mask, extracting prosodic features from the Manchu Latin-transcribed character embedding sequence to generate a prosodic vector, and adjusting the attention weights between characters and speech frames in the pre-trained feature network to obtain the prosodic space, Mel-spectral sequence, and optimal alignment path; finally, inputting the vector into a lightweight multi-band inverse short-time Fourier transform decoder for Manchu to obtain a complete synthesized speech waveform. This invention can significantly and effectively solve the core problem of text-speech frame matching deviation in low-resource Manchu scenarios, and its inference speed is significantly improved compared to general models.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a method and apparatus for Manchu speech synthesis based on character component modeling. Background Technology

[0002] Text-to-speech (TTS) technology, as an important research direction in the field of artificial intelligence, aims to convert text information into natural and fluent speech output. As a core support for human-computer interaction, TTS technology has evolved from traditional Hidden Markov Models (HMMs) to end-to-end text-to-speech (E2E-TTS) models, such as Tacotron, FastSpeech, and Variational Inference with Adversarial Learning for End-to-End Text-to-Speech (VITS).

[0003] In recent years, large-scale models such as NaturalSpeech and Mega-TTS have further improved the synthesis quality of general-purpose languages. However, the synthesis of low-resource languages, such as Manchu, still faces key bottlenecks. As the core carrier of Manchu culture, Manchu has unique linguistic features such as "character component word formation, blurred syllable boundaries, and frequent liaison changes." Currently, annotated Manchu corpora are extremely scarce, and existing E2E-TTS models are all designed for general-purpose languages, making it difficult to adapt to the characteristics of Manchu: on the one hand, general-purpose models such as VITS and FastSpeech2 use phonemes or standardized text as modeling units, which cannot match the mapping rules between Manchu character components and syllables, resulting in syllable fragmentation in synthesized speech; on the other hand, alignment algorithms such as Monotonic AlignmentSearch (MAS) are prone to getting trapped in local optima under low-resource data, making it difficult to handle the complex syllable boundary problems of Manchu.

[0004] Furthermore, decoders such as High-Fidelity Generative Adversarial Networks (HiFi-GAN) suffer from redundant computation and insufficient real-time performance in low-resource hardware environments. This problem is particularly pronounced in Manchu language synthesis scenarios based on Latin-transcribed text, severely limiting the practical application of Manchu speech in cultural dissemination. Therefore, there is an urgent need to design a speech synthesis method adapted to the characteristics of Manchu character components, addressing the language specificity and low-resource nature of Manchu, thereby overcoming the limitations of existing technologies. Summary of the Invention

[0005] To address the technical problem of text-to-speech frame matching discrepancies in low-resource scenarios of Manchu language in existing technologies, this invention provides a Manchu speech synthesis method and apparatus based on character component modeling. The technical solution is as follows:

[0006] On the one hand, a Manchu speech synthesis method based on character component modeling is provided, which is implemented by a Manchu speech synthesis device based on character component modeling. The method includes:

[0007] S1: The original Manchu text is converted according to the transcription rules, then component-level splitting is performed based on the preset Manchu component dictionary, and then cleaned to obtain the cleaned component stream. The transcription rules include transcribing the original Manchu text according to the rules in the Manchu exclusive Latin character table. The transcription process retains 26 English letters, 2 diacritics, and boundary symbol "-". The cleaning includes frequency statistics, correction of biased components, and removal of continuous repeating units.

[0008] S2: Based on the cleaned component stream, a character embedding sequence is formed, a feature template is constructed according to internal rules, and the character embedding sequence and feature template are concatenated in chronological order to obtain a high-dimensional model input tensor. The formation of the character embedding sequence includes assigning a global character-ID to each unique character and mapping the ID to a 64-dimensional vector.

[0009] S3: The high-dimensional model input tensor is fed into the pre-trained duration prediction sub-network to obtain the predicted phoneme duration. The pre-trained duration prediction sub-network is trained based on the feature unit set and includes a stepwise discriminator that adapts the mapping relationship between 1 Manchu component and 1-3 Latin transliteration characters. It is optimized using a double loss function and is trained separately as an independent module in the later stage of the overall model training.

[0010] S4: Based on the predicted phoneme duration, feature extraction is performed on the input tensor of the high-dimensional model to generate a prosodic feature vector. The feature extraction includes first constructing a prosodic mask using a Manchu-specific prosodic dictionary, and then using the prosodic mask to suppress minor characters of unstressed syllables, enhance the attention weights of connected speech boundary characters and word-final stressed syllables.

[0011] S5: Input the character embedding sequence, prosodic mask and predicted phoneme duration into the pre-trained feature extraction network to obtain the prosodic feature space. The pre-trained feature extraction network is trained based on the set of Manchu Latin transcribing components. It adopts a convolution-Transformer dynamic collaborative hybrid structure and includes a hybrid structure of small Transformer blocks with residual connections and 3-5 character local convolutional blocks. The outputs of both are fused by dynamic weights that are adaptively adjusted according to the length of the sequence syllables and the stress position.

[0012] S6: Use a monotonic alignment search algorithm improved with Gaussian noise in the prosodic feature space to search for alignment paths and output the optimal alignment path and the corresponding Mel spectrum sequence.

[0013] S7: Used to load a lightweight multi-band iSTFT decoder for Manchu, perform three-band division and parallel iSTFT processing on the Mel spectrum sequence, combine prosodic feature vectors and optimal alignment paths, and splice waveforms through a Manchu-specific fixed synthesis filter to output a complete synthesized speech waveform.

[0014] Preferably, in step S1, the original Manchu text is converted according to transcription rules, then component-level splitting is performed based on a preset Manchu component dictionary, followed by cleaning to obtain a cleaned component stream. The transcription rules include transcribing the original Manchu text according to the rules in the Manchu-specific Latin character table, retaining 26 English letters, 2 diacritics, and the boundary symbol "-" during the transcription process. The cleaning includes frequency statistics, correction of biased components, and removal of consecutive repeating units, including:

[0015] S11: Based on the preset Manchu-specific Latin character table, the input Manchu original text is Latinized and transcribed, and all punctuation, numbers and non-Manchu symbols are removed to form a preliminary Latin text. The preliminary Latin text consists of 26 English letters, 2 diacritics and the boundary symbol "-".

[0016] S12: The initial Latin text is split into components using the boundary symbol. The text is divided into a list of component sequences using a pre-set Manchu component dictionary. The component splitting includes using the boundary symbol "-" as a component segmentation indicator, so that each splitting unit corresponds to the smallest pronunciation unit of a Manchu phoneme.

[0017] S13: To perform frequency statistics on the component sequence list and correct for biased components, filter out rare components with an occurrence frequency of < 5 to obtain a preliminary component flow;

[0018] S14: Based on the Manchu phonetic corpus, the component stream is subjected to pronunciation matching and correction, and continuous repeating units are removed to obtain the corrected component stream. The pronunciation matching and correction includes correcting spelling deviations to correct pronunciation units.

[0019] S15: Clean the corrected component stream to obtain a cleaned component stream.

[0020] Preferably, in step S2, based on the cleaned component stream, a character embedding sequence is formed. Feature templates are constructed according to internal rules, and the character embedding sequence and feature templates are concatenated in chronological order to obtain a high-dimensional model input tensor. The formation of the character embedding sequence includes assigning a global character-ID to each unique character and mapping the ID to a 64-dimensional vector, including:

[0021] S21: Input the cleaned component stream into a unique character partitioning unit to form a structured character sequence set. The partitioning unit includes lexical segmentation of each character and preservation of its adjacent logical relationships.

[0022] S22: Based on the character-ID allocation mechanism, a unique character-ID and its index identifier are established for each character in the structured character sequence set, and mapped to a 64-dimensional real vector. The character-ID allocation mechanism has cross-corpus consistency and can be mapped to the subsequent vector space. The 64-dimensional real vector includes weight parameters for representing character semantic information and pronunciation features.

[0023] S23: Organize the 64-dimensional real vector into a set of Manchu Latin units, which includes core units covering commonly used Manchu pronunciation components and has virtual character slots and real embedded units.

[0024] S24: Perform consistency verification and quality assessment on the set of Manchu Latin units to obtain the character embedding sequence;

[0025] S25: Based on the acoustic characteristics of Manchu pronunciation, the character-ID, the corresponding 64-dimensional real vector, and the Manchu-specific acoustic feature dimension containing the fundamental frequency and pitch band are concatenated into a feature template according to the feature dimension. The feature template uses component-level character sequences as the basic unit.

[0026] S26: Combine the component stream with the feature template, and concatenate the character-ID, character embedding sequence and feature template according to the feature dimension to form a high-dimensional model input tensor.

[0027] Preferably, in step S3, the high-dimensional model input tensor is fed into a pre-trained duration prediction sub-network to obtain the predicted phoneme duration. The pre-trained duration prediction sub-network is trained based on a set of feature units, includes a stepwise discriminator that adapts the mapping relationship between one Manchu component and 1-3 Latin transliteration characters, is optimized using a double loss function, and is trained separately as an independent module in the later stages of the overall model training. It includes:

[0028] S31: Obtain the Manchu text library, first transcribe the Manchu text into Latin, and then clean it to obtain the cleaned text;

[0029] S32: Using boundary symbols to identify and use a Manchu component dictionary, the cleaned text is split into components at the component level to obtain a component set. The Manchu component dictionary includes 66 component-Latin mappings.

[0030] S33: Perform frequency statistics on the component set, delete low-frequency components that appear less than 5 times, then perform pronunciation matching and correct deviations based on the speech corpus, remove duplicates from the verified components, and obtain training data containing training units;

[0031] S34: Treat each training unit in the training data as a component-level character sequence, assign a unique index to each character appearing in the sequence, and map the index to a 64-dimensional vector to form a character embedding matrix;

[0032] S35: Calculate the logarithmic duration of each training unit in the speech corpus, and construct prosodic features based on the hidden representation generated by the encoder;

[0033] S36: Construct a duration prediction subnetwork and input prosodic features into it. Through adversarial training, make the duration prediction distribution conform to the duration characteristics of real Manchu speech. The adversarial training includes using a GAN framework, treating GAN-DP as an independent module, with a stepwise discriminator that can independently evaluate the variable duration of each Latin transcribing character. During training, a double loss function is used to complete gradient backpropagation. In the later stages of overall model training, only the duration prediction subnetwork is trained separately, and the weights of the duration prediction subnetwork are iteratively updated. After training, the complete duration prediction subnetwork and character embedding matrix are saved to permanent storage for inference.

[0034] Preferably, step S4 treats each training unit in the training data as a component-level character sequence, assigns a unique index to each character appearing in the sequence, and maps the index to a 64-dimensional vector to form a character embedding matrix, including:

[0035] S41: Construct a prosodic mask using a pre-trained Manchu-specific prosodic dictionary. This mask identifies the positions of stressed syllables, liaison boundary characters, and word-final stressed syllables on the time axis of the character embedding sequence, while also marking the positions of minor characters in unstressed syllables.

[0036] S42: The attention weight of minor characters of unstressed syllables in the character embedding sequence is suppressed by the attention weighting unit, and the suppression is implemented in the corresponding attention weighting unit to reduce its interference with prosodic feature extraction;

[0037] S43: The weighted character embedding sequence is input into the feature aggregation module, and the feature is aggregated through linear mapping and normalization operations to obtain the prosodic feature vector.

[0038] Preferably, in step S5, the character embedding sequence, prosodic mask, and predicted phoneme duration are input into a pre-trained feature extraction network to obtain a prosodic feature space. The pre-trained feature extraction network is trained based on a set of Manchu-Latin transcribing components, employing a convolutional-Transformer dynamic collaborative hybrid structure. It includes a hybrid structure of small Transformer blocks with residual connections and 3-5 character local convolutional blocks. The outputs of both are fused through dynamic weights that adaptively adjust with the syllable length and stress position of the sequence, including:

[0039] S51: Obtain the Manchu text library, first transcribe the Manchu text into Latin, and then clean it to obtain the cleaned text;

[0040] S52: After cleaning, the text is subjected to character filtering, component-level splitting, low-frequency component removal, pronunciation matching correction and deduplication, and then labeled to obtain labeled training data;

[0041] S53: Based on labeled training data, a Manchu-specific prosodic dictionary is trained;

[0042] S54: Based on the Manchu-specific prosodic dictionary, a dynamic attention mask is constructed for the character sequences in the labeled training data, and a convolutional block is designed to extract local feature maps. A small Transformer block with residual connections is introduced to perform self-attention. The outputs of the two are fused by dynamic weights that are adaptively adjusted according to the length of the syllables and the stress position of the sequence to obtain a comprehensive prosodic feature representation.

[0043] S55: Persist the convolutional blocks, Transformer blocks, dynamic attention masks and their weights together to form a pre-trained feature extraction network.

[0044] Preferably, the S7, used to load the lightweight multi-band iSTFT decoder for Manchu, performs three-band division and parallel iSTFT processing on the Mel spectrum sequence, combines prosodic feature vectors and optimal alignment paths, and splices waveforms through a Manchu-specific fixed synthesis filter to output a complete synthesized speech waveform, including:

[0045] S71: The Manchu lightweight multi-band inverse short-time Fourier transform decoder first divides the Mel spectrum sequence into three dedicated frequency bands based on the spectral distribution of Manchu vowels, unvoiced consonants, and voiceless consonants. The division includes a low-frequency band, a mid-frequency band, and a high-frequency band. The spectral distribution of Manchu vowels is 200-800Hz, the spectral distribution of unvoiced consonants is 800-3000Hz, and the spectral distribution of voiceless consonants is 3000-5000Hz.

[0046] S72: The three dedicated frequency bands are each configured with an independent inverse short-time Fourier transform module, which generates corresponding time-domain waveform segments in parallel;

[0047] S73: Input the corresponding time-domain waveform segment, and splice it through the frequency band fusion module based on prosodic feature vector and optimal alignment path to obtain a complete synthesized speech waveform. The frequency band fusion module includes a fixed synthesis filter. The Manchu lightweight multi-band inverse short-time Fourier transform decoder uses iSTFT to replace the redundant convolution operation of HiFi-GAN.

[0048] On the other hand, a Manchu speech synthesis device based on character component modeling is provided. This device is applied to a Manchu speech synthesis method based on character component modeling. The device includes:

[0049] The data preprocessing module is used to convert the original Manchu text according to the transcription rules, then perform component-level splitting based on a pre-set Manchu component dictionary, and then clean it to obtain a cleaned component stream. The transcription rules include transcribing the original Manchu text according to the rules in the Manchu-specific Latin character table. The transcription process retains 26 English letters, 2 diacritics, and the boundary symbol "-". The cleaning includes frequency statistics, correction of biased components, and removal of consecutive repeating units.

[0050] Feature template module: used to form a character embedding sequence based on the cleaned component stream, construct feature templates according to internal rules, and concatenate the character embedding sequence and feature templates in chronological order to obtain a high-dimensional model input tensor. The formation of the character embedding sequence includes assigning a global character-ID to each unique character and mapping the ID to a 64-dimensional vector.

[0051] Duration prediction module: It is used to feed the input tensor of the high-dimensional model into the pre-trained duration prediction sub-network to obtain the predicted phoneme duration. The pre-trained duration prediction sub-network is trained based on the feature unit set and contains a stepwise discriminator that adapts the mapping relationship between 1 Manchu component and 1-3 Latin transliteration characters. It is optimized with a double loss function and is trained separately as an independent module in the later stage of the overall model training.

[0052] Prosodic feature module: used to extract features from the input tensor of the high-dimensional model based on the predicted phoneme duration, and generate prosodic feature vector. The feature extraction includes first constructing a prosodic mask using a Manchu-specific prosodic dictionary, and then using the prosodic mask to suppress minor characters of unstressed syllables, enhance the attention weights of connected speech boundary characters and word-final stressed syllables.

[0053] Feature extraction module: used to input character embedding sequence, prosodic mask and predicted phoneme duration into pre-trained feature extraction network to obtain prosodic feature space. The pre-trained feature extraction network is trained based on Manchu Latin transcribing component set, adopts convolution-Transformer dynamic collaborative hybrid structure, and includes a hybrid structure of small Transformer blocks with residual connections and 3-5 character local convolution blocks. The outputs of both are fused by dynamic weights that are adaptively adjusted according to the syllable length and stress position of the sequence.

[0054] Alignment Path Module: Used to perform alignment path search in prosodic feature space using a monotonic alignment search algorithm improved with Gaussian noise, and output the optimal alignment path and the corresponding Mel spectrum sequence;

[0055] The speech synthesis module loads a lightweight multi-band iSTFT decoder for Manchu, performs three-band division and parallel iSTFT processing on the Mel spectrum sequence, combines prosodic feature vectors and the optimal alignment path, and splices waveforms through a Manchu-specific fixed synthesis filter to output a complete synthesized speech waveform.

[0056] On the other hand, a Manchu speech synthesis device based on character component modeling is provided. The Manchu speech synthesis device based on character component modeling includes: a processor and a memory. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they implement the method described in any of the above-described methods for Manchu speech synthesis based on character component modeling.

[0057] On the other hand, a computer-readable storage medium is provided, characterized in that program code is stored in the computer-readable storage medium.

[0058] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0059] This invention effectively solves the core problem of text-to-speech frame matching deviation in low-resource Manchu scenarios. Compared with the general VITS model, it has the following advantages: the alignment deviation rate and character error rate are reduced, while the accuracy of connected speech sound change restoration is significantly improved, and the naturalness is close to the level of human Manchu pronunciation. The Mean Opinion Score (MOS) of the synthesized speech is significantly improved. Finally, this invention adopts a lightweight multi-band inverse short-time Fourier transform decoder for Manchu, and by replacing the redundant convolution operations of HiFi-GAN with iSTFT and combining it with parallel processing of Manchu-specific frequency bands, the inference speed is significantly improved compared with the general model. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of a Manchu speech synthesis method based on character component modeling provided by an embodiment of the present invention;

[0062] Figure 2 This is a diagram of a Manchu-specific GAN-DP architecture provided in an embodiment of the present invention;

[0063] Figure 3 This is a block diagram of a Manchu speech synthesis device based on character component modeling provided in an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of the structure of a Manchu speech synthesis device based on character component modeling provided in an embodiment of the present invention. Detailed Implementation

[0065] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0066] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0067] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0068] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0069] Terminology Explanation:

[0070] iSTFT decoder: Inverse Short Time Fourier Transform decoder, reconstructs time-domain audio waveforms from frequency domain features.

[0071] Transformer: A basic sequence modeling unit containing self-attention and feedforward networks.

[0072] GAN DP: Duration Predictor module within the Generative Adversarial Network framework.

[0073] HiFi-GAN: A high-fidelity GAN vocoder used for converting Mel spectrum to high-quality waveforms.

[0074] Monotonic Alignment Search (MAS): This search method finds the monotonic optimal alignment between text and speech frames.

[0075] VITS Model: Variational inference adversarial end-to-end TTS model for direct text generation of highly natural speech.

[0076] Dual loss function (Ladv+LMSE): adversarial loss + mean squared error loss, balancing perception quality and reconstruction accuracy.

[0077] Duration predictor: Estimates the length of the speech frame corresponding to each phoneme in the text.

[0078] Duration discriminator: A duration discriminator that distinguishes between the actual and predicted phoneme duration distributions.

[0079] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0080] This invention provides a Manchu speech synthesis method based on character component modeling. This method can be implemented by a Manchu speech synthesis device based on character component modeling, which can be a terminal or a server. Figure 1 The flowchart shown is for a Manchu speech synthesis method based on character component modeling. The processing flow of this method may include the following steps:

[0081] S1: The original Manchu text is converted according to the transcription rules, then component-level splitting is performed based on the preset Manchu component dictionary, and then cleaned to obtain the cleaned component stream. The transcription rules include transcribing the original Manchu text according to the rules in the Manchu exclusive Latin character table. The transcription process retains 26 English letters, 2 diacritics, and boundary symbol "-". The cleaning includes frequency statistics, correction of biased components, and removal of continuous repeating units.

[0082] Preferably, S1 includes:

[0083] S11: Based on the preset Manchu-specific Latin character table, the input Manchu original text is Latinized and transcribed, and all punctuation, numbers and non-Manchu symbols are removed to form a preliminary Latin text. The preliminary Latin text consists of 26 English letters, 2 diacritics and the boundary symbol "-".

[0084] S12: The initial Latin text is split into components using the boundary symbol. The text is divided into a list of component sequences using a pre-set Manchu component dictionary. The component splitting includes using the boundary symbol "-" as a component segmentation indicator, so that each splitting unit corresponds to the smallest pronunciation unit of a Manchu phoneme.

[0085] S13: To perform frequency statistics on the component sequence list and correct for biased components, filter out rare components with an occurrence frequency of < 5 to obtain a preliminary component flow;

[0086] S14: Based on the Manchu phonetic corpus, the component stream is subjected to pronunciation matching and correction, and continuous repeating units are removed to obtain the corrected component stream. The pronunciation matching and correction includes correcting spelling deviations to correct pronunciation units.

[0087] S15: Clean the corrected component stream to obtain a cleaned component stream.

[0088] It should be noted that Manchu belongs to the Altaic language family and is an agglutinative language, with the following significant characteristics: Writing system: Manchu is a phonetic script, with a non-one-to-one correspondence between characters and phonetic units (one Manchu character component can correspond to 1-3 phonetic units), and the rules for connecting characters are complex; directly splitting Manchu characters can easily lead to the fragmentation of phonetic units. Phonetic features: It is mainly composed of polysyllabic words, with vowel assimilation and consonant dropouts between syllables, and its prosody depends on the coordination of the overall pronunciation sequence. Low resource characteristics: Existing Manchu phonetic and textual data are scarce, making it difficult to support direct modeling of the complex Manchu character system.

[0089] It should be further noted that under low-resource data conditions, the model has difficulty fully learning the prosodic features of Manchu connected speech, such as sound changes and stress rhythm. Furthermore, the convolutional structure of the general model has limited ability to capture long-distance dependencies, resulting in insufficient fluency of synthesized speech.

[0090] It should be further explained that in some embodiments, the mapping relationship between Manchu character components and syllables is complex, with one Manchu character component corresponding to 1-3 speech units. General alignment algorithms cannot adapt to this special mapping, resulting in deviations in the matching of text and speech frames.

[0091] Preferably, the Manchu script is first transcribed into Latin, and then the Latin character set is extracted from the transcription result as the training unit. The reasons are as follows: (1) The Latin transcription rules have transformed the complex correspondence between Manchu character components and speech units (1-3 speech units) into an intuitive Latin character sequence, avoiding the breakage of speech units caused by directly splitting Manchu characters, and are more in line with the actual pronunciation logic; (2) The Latin character set has a simple structure (composed of 26 basic letters and a small number of diacritics), which can reduce the difficulty of the model to learn the character-speech mapping relationship under low resource data and improve training efficiency; (3) The Latin transcription completely preserves the prosodic information such as Manchu liaison, phonological changes, and stress rhythm, which is convenient for the model to learn long-distance speech dependencies.

[0092] It should be noted that the original text or the Manchu-Latin transliteration text library may contain punctuation marks, numbers, and non-Manchu symbols, which require subsequent cleaning steps. Example text: " After cleaning, the result is "manju bita-". This step removes irrelevant interference and retains the core valid sequences of the Manchu Latin transliteration. The Manchu-specific Latin character table explicitly stipulates that only 26 English letters, 2 diacritics, and 1 boundary symbol "-" should be retained. This step provides a selection criterion for subsequent text cleaning, ensuring that only valid characters related to Manchu pronunciation and word formation are processed. Using boundary symbol recognition and a pre-trained Manchu component dictionary (containing 66 component-Latin mappings), the cleaned text is decomposed into component-level segments. Taking "manju bita-" as an example, it is decomposed into component-level character sequences such as "ma", "n", "ju", "bi", and "ta". This step decomposes continuous text into finer-grained components corresponding to Manchu pronunciation units, facilitating subsequent model learning. Frequency statistics are performed to filter out components with a frequency less than 5 to avoid overfitting the model due to obscure units; then, pronunciation matching is performed by comparing with speech corpora to correct potential deviations (such as misinterpreting "juu"). The value was corrected to "ju"), and duplicate elements were removed from the verified elements. This step yields high-quality elements suitable for modeling.

[0093] S2: Based on the cleaned component stream, a character embedding sequence is formed, a feature template is constructed according to internal rules, and the character embedding sequence and feature template are concatenated in chronological order to obtain a high-dimensional model input tensor. The formation of the character embedding sequence includes assigning a global character-ID to each unique character and mapping the ID to a 64-dimensional vector.

[0094] Preferably, S2 includes:

[0095] S21: Input the cleaned component stream into a unique character partitioning unit to form a structured character sequence set. The partitioning unit includes lexical segmentation of each character and preservation of its adjacent logical relationships.

[0096] S22: Based on the character-ID allocation mechanism, a unique character-ID and its index identifier are established for each character in the structured character sequence set, and mapped to a 64-dimensional real vector. The character-ID allocation mechanism has cross-corpus consistency and can be mapped to the subsequent vector space. The 64-dimensional real vector includes weight parameters for representing character semantic information and pronunciation features.

[0097] S23: Organize the 64-dimensional real vector into a set of Manchu Latin units, which includes core units covering commonly used Manchu pronunciation components and has virtual character slots and real embedded units.

[0098] S24: Perform consistency verification and quality assessment on the set of Manchu Latin units to obtain the character embedding sequence;

[0099] S25: Based on the acoustic characteristics of Manchu pronunciation, the character-ID, the corresponding 64-dimensional real vector, and the Manchu-specific acoustic feature dimension containing the fundamental frequency and pitch band are concatenated into a feature template according to the feature dimension. The feature template uses component-level character sequences as the basic unit.

[0100] S26: Combine the component stream with the feature template, and concatenate the character-ID, character embedding sequence and feature template according to the feature dimension to form a high-dimensional model input tensor.

[0101] In some embodiments, character embedding encoding is performed on each modeling unit. First, a unique index is assigned to each character, and then the index is converted into a 64-dimensional vector representation. This vector representation maps the semantic and phonetic information of the character to a low-dimensional space, facilitating model computation and learning. Simultaneously, the 64-dimensional embedding dimension effectively balances representational power and computational cost in low-resource scenarios. The final set of Manchu Latin transliteration training units, such as {“ma”, “n”, “ju”, “bi”, “ta”} in the example, contains core units covering commonly used Manchu phonetic components. These units undergo rigorous screening and validation through data preprocessing, providing high-quality training data for subsequent Manchu speech synthesis models and ensuring model performance in low-resource scenarios. Through this design, the core adaptation problem of Manchu speech synthesis is solved at the underlying modeling unit level, laying the foundation for subsequent triple optimization of "alignment accuracy-naturalness-efficiency".

[0102] S3: The high-dimensional model input tensor is fed into the pre-trained duration prediction sub-network to obtain the predicted phoneme duration. The pre-trained duration prediction sub-network is trained based on the feature unit set and includes a stepwise discriminator that adapts the mapping relationship between 1 Manchu component and 1-3 Latin transliteration characters. It is optimized using a double loss function and is trained separately as an independent module in the later stage of the overall model training.

[0103] Preferably, S3 includes:

[0104] S31: Obtain the Manchu text library, first transcribe the Manchu text into Latin, and then clean it to obtain the cleaned text;

[0105] S32: Using boundary symbols to identify and use a Manchu component dictionary, the cleaned text is split into components at the component level to obtain a component set. The Manchu component dictionary includes 66 component-Latin mappings.

[0106] S33: Perform frequency statistics on the component set, delete low-frequency components that appear less than 5 times, then perform pronunciation matching and correct deviations based on the speech corpus, remove duplicates from the verified components, and obtain training data containing training units;

[0107] S34: Treat each training unit in the training data as a component-level character sequence, assign a unique index to each character appearing in the sequence, and map the index to a 64-dimensional vector to form a character embedding matrix;

[0108] S35: Calculate the logarithmic duration of each training unit in the speech corpus, and construct prosodic features based on the hidden representation generated by the encoder;

[0109] S36: Construct a duration prediction subnetwork and input prosodic features into it. Through adversarial training, make the duration prediction distribution conform to the duration characteristics of real Manchu speech. The adversarial training includes using a GAN framework, with GAN-DP as an independent module. Its discriminator is a stepwise discriminator that can independently evaluate the variable duration of each Latin transcribing character. During training, a double loss function is used to complete gradient backpropagation. In the later stage of the overall model training, only the duration prediction subnetwork is trained separately, and the weights of the duration prediction subnetwork are iteratively updated. After training, the complete duration prediction subnetwork and character embedding matrix are saved to permanent storage for inference.

[0110] In some embodiments, the input is "logarithmic duration of Manchu Latin transliteration characters + text hidden layer features," which allows for independent evaluation of the variable duration of each transliteration character, overcoming the limitations of fixed-length input. Considering the unique characteristics of Manchu Latin transliteration text, the dimensions of the text hidden layer features are expanded to accommodate the mapping relationship of "1 Manchu original character component corresponding to 1-3 Latin transliteration characters," improving the modeling accuracy of duration changes caused by liaison in Manchu polysyllabic words.

[0111] It should be noted that the random duration predictor is optimized using a dual loss function: the least squares loss (Ladv) function and the mean squared error loss (LMSE) function derived from adversarial learning. Adversarial training optimizes the duration prediction distribution, making the prediction results more closely reflect the duration characteristics of authentic Manchu speech. The formula is as follows:

[0112] Adversarial loss (discriminator D):

[0113]

[0114] Adversarial loss (generator G):

[0115]

[0116] Mean Squared Error Loss (MSE Loss):

[0117]

[0118] in, d The actual duration of the Manchu Latin transliteration characters. G ( z d , h text ) represents the generator prediction duration. z d It is a random noise vector. h text This is a hidden feature of the Manchu-Latin transliteration text.

[0119] It needs to be further explained that, such as Figure 2As shown, GAN-DP was trained separately as an independent module in the later stage of model training. The results showed that compared with the general VITS, the training time was reduced and the duration prediction error was significantly reduced, especially the prediction accuracy in the phonological change interval of polysyllabic words was significantly improved. This optimization is due to the two Manchu-specific designs of GAN-DP: (1) the discriminator input is adapted to the mapping relationship of "1 Manchu component corresponds to 1-3 Latin transliteration characters", which can independently model the variable duration of each transliteration character; (2) the dual loss function (Ladv+LMSE) is optimized to make the prediction distribution more in line with the duration characteristics of real Manchu speech (such as the higher duration of stressed syllables in polysyllabic words).

[0120] S4: Based on the predicted phoneme duration, feature extraction is performed on the input tensor of the high-dimensional model to generate a prosodic feature vector. The feature extraction includes first constructing a prosodic mask using a Manchu-specific prosodic dictionary, and then using the prosodic mask to suppress minor characters of unstressed syllables, enhance the attention weights of connected speech boundary characters and word-final stressed syllables.

[0121] Preferably, S4 includes:

[0122] S41: Construct a prosodic mask using a pre-trained Manchu-specific prosodic dictionary. This mask identifies the positions of stressed syllables, liaison boundary characters, and word-final stressed syllables on the time axis of the character embedding sequence, while also marking the positions of minor characters in unstressed syllables.

[0123] S42: The attention weight of minor characters of unstressed syllables in the character embedding sequence is suppressed by the attention weighting unit, and the suppression is implemented in the corresponding attention weighting unit to reduce its interference with prosodic feature extraction;

[0124] S43: The weighted character embedding sequence is input into the feature aggregation module, and the feature is aggregated through linear mapping and normalization operations to obtain the prosodic feature vector.

[0125] In some embodiments, the convolutional block is responsible for capturing local speech patterns of 3-5 consecutive Latin transliterated characters, such as the basic spelling rules of Manchu consonants and vowels, and the co-pronunciation of adjacent characters; the Transformer block aggregates long-distance information of more than 5 characters through a self-attention mechanism, focusing on modeling the phonological changes of connected speech within multisyllabic words and the prosodic coherence across word boundaries; the outputs of both are fused through dynamic weights: the weights are adaptively adjusted according to the syllable length and stress position of the current processing sequence. Compared with traditional models, this design improves the accuracy of capturing Manchu-specific prosodic features by more than 20%, and also improves the naturalness of the synthesized speech, especially solving the problems of the abruptness after connected speech of multisyllabic words and the stress position shift.

[0126] S5: Input the character embedding sequence, prosodic mask and predicted phoneme duration into the pre-trained feature extraction network to obtain the prosodic feature space. The pre-trained feature extraction network is trained based on the set of Manchu Latin transcribing components. It adopts a convolution-Transformer dynamic collaborative hybrid structure and includes a hybrid structure of small Transformer blocks with residual connections and 3-5 character local convolutional blocks. The outputs of both are fused by dynamic weights that are adaptively adjusted according to the length of the sequence syllables and the stress position.

[0127] Preferably, S5 includes:

[0128] S51: Obtain the Manchu text library, first transcribe the Manchu text into Latin, and then clean it to obtain the cleaned text;

[0129] S52: After cleaning, the text is subjected to character filtering, component-level splitting, low-frequency component removal, pronunciation matching correction and deduplication, and then labeled to obtain labeled training data;

[0130] S53: Based on labeled training data, a Manchu-specific prosodic dictionary is trained;

[0131] S54: Based on the Manchu-specific prosodic dictionary, a dynamic attention mask is constructed for the character sequences in the labeled training data, and a convolutional block is designed to extract local feature maps. A small Transformer block with residual connections is introduced to perform self-attention. The outputs of the two are fused by dynamic weights that are adaptively adjusted according to the length of the syllables and the stress position of the sequence to obtain a comprehensive prosodic feature representation.

[0132] S55: Persist the convolutional blocks, Transformer blocks, dynamic attention masks and their weights together to form a pre-trained feature extraction network.

[0133] S6: Use a monotonic alignment search algorithm improved with Gaussian noise in the prosodic feature space to search for alignment paths and output the optimal alignment path and the corresponding Mel spectrum sequence.

[0134] In some embodiments, to address the problem that traditional MAS is prone to getting trapped in local optima due to the complexity of character-speech mapping under low-resource Manchu data, a dynamic Gaussian noise adjustment mechanism is introduced to optimize alignment accuracy.

[0135] In some embodiments, to enhance the robustness of the model, a Gaussian noise term ε is added to the alignment ratio calculation during training, where ε represents the alignment ratio between sequence position i and frequency position j. The improved alignment ratio formula is as follows:

[0136]

[0137] This represents the alignment ratio between positions i and j, which is calculated based on values ​​obtained from the standard normal distribution. The formula for calculating the basic alignment probability is as follows:

[0138]

[0139] Where i represents the position of the Manchu character component sequence, and j represents the position of the speech frame. z j The latent variable is the output of the normalized stream. μ i , σ i The mean and variance of the Manchu component.

[0140] The noise term ε is generated by "standard normal distribution sampling noise × standard deviation of Pi × noise scaling factor". Considering the low resource content and complexity of character-speech mapping in Manchu, the initial scaling factor is set to 0.01 (higher than 0.001 for common languages) to enhance the exploration of ambiguous alignment boundaries. It is dynamically adjusted during training, decreasing by 2 × 10-6 every 1000 training steps until converging to 0.001, balancing exploration capability and training stability. In the Manchu-Latin transcribing text test set, this mechanism significantly reduced the proportion of character-speech frame alignment deviations, laying the foundation for accurate capture of subsequent prosodic features.

[0141] S7: Used to load a lightweight multi-band iSTFT decoder for Manchu, perform three-band division and parallel iSTFT processing on the Mel spectrum sequence, combine prosodic feature vectors and optimal alignment paths, and splice waveforms through a Manchu-specific fixed synthesis filter to output a complete synthesized speech waveform.

[0142] Preferably, S7 includes:

[0143] S71: The Manchu lightweight multi-band inverse short-time Fourier transform decoder first divides the Mel spectrum sequence into three dedicated frequency bands based on the spectral distribution of Manchu vowels, unvoiced consonants, and voiceless consonants. The division includes a low-frequency band, a mid-frequency band, and a high-frequency band. The spectral distribution of Manchu vowels is 200-800Hz, the spectral distribution of unvoiced consonants is 800-3000Hz, and the spectral distribution of voiceless consonants is 3000-5000Hz.

[0144] S72: The three dedicated frequency bands are each configured with an independent inverse short-time Fourier transform module, which generates corresponding time-domain waveform segments in parallel;

[0145] S73: Input the corresponding time-domain waveform segment, and splice it through the frequency band fusion module based on prosodic feature vector and optimal alignment path to obtain a complete synthesized speech waveform. The frequency band fusion module includes a fixed synthesis filter. The Manchu lightweight multi-band inverse short-time Fourier transform decoder uses iSTFT to replace the redundant convolution operation of HiFi-GAN.

[0146] A lightweight multi-band inverse short-time Fourier transform decoder for Manchu uses an inverse short-time Fourier transform (iSTFT) as the final output module for speech synthesis. To address the computational redundancy and poor spectral adaptability of general HiFi-GAN decoders, an "iSTFT + multi-band parallel processing" architecture is adopted. This architecture is optimized based on the spectral characteristics of Manchu. Through spectral analysis of 1000 Manchu Latin-transcribed corpora, and according to the spectral distribution patterns of Manchu vowels and consonants (vowel core spectrum 200-800Hz, unvoiced consonants concentrated 800-3000Hz, voiceless consonants high-frequency details 3000-5000Hz), the frequency bands are divided into three dedicated intervals. Starting directly from the target feature sequence (Mel spectrogram) output earlier, three independent iSTFT modules process different frequency bands in parallel: each module generates a time-domain waveform segment only for its corresponding frequency band, avoiding the redundancy of serial calculation across the entire frequency band in traditional decoders. Finally, a frequency band fusion module concatenates the segments to generate a complete time-domain speech signal. Compared to general HiFi-GAN, this solution significantly reduces inference latency and improves speed by nearly 50% when processing Manchu Latin-transcribed text of 50 characters or more, making it suitable for real-time applications on low-resource hardware. Addressing the timbre shift issue caused by general decoders' incompatibility with Manchu spectral characteristics, this solution employs a "Manchu-specific frequency band division + fixed synthesis filter" design. This ensures the filter's passband center frequency precisely matches the unique spectral peaks of Manchu vowels, fundamentally suppressing interference from non-target frequency band noise on the core Manchu timbre. Simultaneously, the multi-band parallel processing mechanism fully preserves unique Manchu phonetic details such as phonological changes in connected speech and the continuity of final vowels, avoiding prosodic breaks caused by the coarse processing across all frequency bands in general decoders, thus achieving accurate reproduction of Manchu-specific timbre.

[0147] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0148] Figure 3 This is a block diagram illustrating a Manchu speech synthesis apparatus based on character component modeling, according to an exemplary embodiment. The apparatus is used in a Manchu speech synthesis method based on character component modeling. (Refer to...) Figure 3 The device includes a data preprocessing module, a feature template module, a duration prediction module, a prosodic feature module, a feature extraction module, an alignment path module, and a speech synthesis module.

[0149] The data preprocessing module is used to convert the original Manchu text according to the transcription rules, then perform component-level splitting based on a pre-set Manchu component dictionary, and then clean it to obtain a cleaned component stream. The transcription rules include transcribing the original Manchu text according to the rules in the Manchu-specific Latin character table. The transcription process retains 26 English letters, 2 diacritics, and the boundary symbol "-". The cleaning includes frequency statistics, correction of biased components, and removal of consecutive repeating units.

[0150] Feature template module: used to form a character embedding sequence based on the cleaned component stream, construct feature templates according to internal rules, and concatenate the character embedding sequence and feature templates in chronological order to obtain a high-dimensional model input tensor. The formation of the character embedding sequence includes assigning a global character-ID to each unique character and mapping the ID to a 64-dimensional vector.

[0151] Duration prediction module: It is used to feed the input tensor of the high-dimensional model into the pre-trained duration prediction sub-network to obtain the predicted phoneme duration. The pre-trained duration prediction sub-network is trained based on the feature unit set and contains a stepwise discriminator that adapts the mapping relationship between 1 Manchu component and 1-3 Latin transliteration characters. It is optimized with a double loss function and is trained separately as an independent module in the later stage of the overall model training.

[0152] Prosodic feature module: used to extract features from the input tensor of the high-dimensional model based on the predicted phoneme duration, and generate prosodic feature vector. The feature extraction includes first constructing a prosodic mask using a Manchu-specific prosodic dictionary, and then using the prosodic mask to suppress minor characters of unstressed syllables, enhance the attention weights of connected speech boundary characters and word-final stressed syllables.

[0153] Feature extraction module: used to input character embedding sequence, prosodic mask and predicted phoneme duration into pre-trained feature extraction network to obtain prosodic feature space. The pre-trained feature extraction network is trained based on Manchu Latin transcribing component set, adopts convolution-Transformer dynamic collaborative hybrid structure, and includes a hybrid structure of small Transformer blocks with residual connections and 3-5 character local convolution blocks. The outputs of both are fused by dynamic weights that are adaptively adjusted according to the syllable length and stress position of the sequence.

[0154] Alignment Path Module: Used to perform alignment path search in prosodic feature space using a monotonic alignment search algorithm improved with Gaussian noise, and output the optimal alignment path and the corresponding Mel spectrum sequence;

[0155] The speech synthesis module loads a lightweight multi-band iSTFT decoder for Manchu, performs three-band division and parallel iSTFT processing on the Mel spectrum sequence, combines prosodic feature vectors and the optimal alignment path, and splices waveforms through a Manchu-specific fixed synthesis filter to output a complete synthesized speech waveform.

[0156] Figure 4 This is a schematic diagram of the structure of a Manchu speech synthesis device based on character component modeling provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the Manchu speech synthesis device based on character component modeling can include the above-mentioned... Figure 3 The illustrated Manchu speech synthesis device is based on character component modeling. Optionally, the Manchu speech synthesis device 410 based on character component modeling may include a first processor 2001.

[0157] Optionally, the Manchu speech synthesis device 410 based on character component modeling may also include a memory 2002 and a transceiver 2003.

[0158] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0159] The following is combined with Figure 4 A detailed introduction to each component of the Manchu speech synthesis device 410 based on character component modeling is provided below:

[0160] The first processor 2001 is the control center of the Manchu speech synthesis device 410 based on character component modeling. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0161] Optionally, the first processor 2001 can perform various functions of the Manchu speech synthesis device 410 based on character component modeling by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0162] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0163] In a specific implementation, as one example, the Manchu speech synthesis device 410 based on character component modeling may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0164] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0165] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the Manchu speech synthesis device 410 modeled based on character components. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0166] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0167] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0168] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the Manchu speech synthesis device 410 based on character component modeling. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0169] It should be noted that, Figure 4 The structure of the Manchu speech synthesis device 410 based on character component modeling shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0170] Furthermore, the technical effects of the Manchu speech synthesis device 410 based on character component modeling can be referred to the technical effects of the Manchu speech synthesis method based on character component modeling described in the above method embodiments, and will not be repeated here.

[0171] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0172] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0173] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0174] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0175] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0176] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0177] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0179] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0182] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A Manchu speech synthesis method based on character component modeling, characterized in that, The method includes: S1: The original Manchu text is converted according to the transcription rules, then component-level splitting is performed based on the preset Manchu component dictionary, and then cleaned to obtain the cleaned component stream. The transcription rules include transcribing the original Manchu text according to the rules in the Manchu exclusive Latin character table. The transcription process retains 26 English letters, 2 diacritics, and boundary symbol "-". The cleaning includes frequency statistics, correction of biased components, and removal of continuous repeating units. S2: Based on the cleaned component stream, a character embedding sequence is formed. A feature template is constructed according to internal rules, and the character embedding sequence and feature template are concatenated in chronological order to obtain a high-dimensional model input tensor. The formation of the character embedding sequence includes assigning a global character-ID to each unique character and mapping the ID to a 64-dimensional vector. The construction of the feature template according to internal rules includes concatenating the character-ID, the corresponding 64-dimensional real vector, and the Manchu-specific acoustic feature dimension containing the fundamental frequency and pitch band according to the feature dimension to form a feature template. The Manchu-specific acoustic feature dimension containing the fundamental frequency and pitch band is set according to the acoustic characteristics of Manchu pronunciation. S3: The high-dimensional model input tensor is fed into the pre-trained duration prediction sub-network to obtain the predicted phoneme duration. The pre-trained duration prediction sub-network is trained based on the feature unit set and contains a stepwise discriminator that adapts the mapping relationship between 1 Manchu component and 1-3 Latin transliteration characters. It is optimized using a double loss function and is trained separately as an independent module in the later stage of the overall model training. S4: Based on the predicted phoneme duration, feature extraction is performed on the input tensor of the high-dimensional model to generate a prosodic feature vector. The feature extraction includes first constructing a prosodic mask using a Manchu-specific prosodic dictionary, and then using the prosodic mask to suppress minor characters of unstressed syllables, enhance the attention weights of connected speech boundary characters and word-final stressed syllables. S5: Input the character embedding sequence, prosodic mask and predicted phoneme duration into the pre-trained feature extraction network to obtain the prosodic feature space. The pre-trained feature extraction network is trained based on the set of Manchu Latin transcribing components. It adopts a convolution-Transformer dynamic collaborative hybrid structure and includes a hybrid structure of small Transformer blocks with residual connections and 3-5 character local convolutional blocks. The outputs of both are fused by dynamic weights that are adaptively adjusted according to the length of the sequence syllables and the stress position. S6: Use a monotonic alignment search algorithm improved with Gaussian noise in the prosodic feature space to search for alignment paths and output the optimal alignment path and the corresponding Mel spectrum sequence. S7: Used to load a lightweight multi-band iSTFT decoder for Manchu, perform three-band division and parallel iSTFT processing on the Mel spectrum sequence, combine prosodic feature vectors and optimal alignment paths, and output a complete synthesized speech waveform by splicing waveforms through a Manchu-specific fixed synthesis filter. The passband center frequency of the Manchu-specific fixed synthesis filter is set according to the spectral distribution of Manchu vowels, unvoiced consonants and voiceless consonants. The low-frequency passband corresponds to the Manchu vowel spectrum distribution of 200-800Hz, the mid-frequency passband corresponds to the unvoiced consonant spectrum distribution of 800-3000Hz, and the high-frequency passband corresponds to the voiceless consonant spectrum distribution of 3000-5000Hz.

2. The Manchu speech synthesis method based on character component modeling according to claim 1, characterized in that, S1 converts the original Manchu text according to transcription rules, then performs component-level splitting based on a pre-set Manchu component dictionary, and then cleans it to obtain a cleaned component stream. The transcription rules include transcribing the original Manchu text according to the rules in the Manchu-specific Latin character table, retaining 26 English letters, 2 diacritics, and the boundary symbol "-" during the transcription process. The cleaning includes frequency statistics, correction of biased components, and removal of consecutive repeating units, including: S11: Based on the preset Manchu-specific Latin character table, the input Manchu original text is Latinized and transcribed, and all punctuation, numbers and non-Manchu symbols are removed to form a preliminary Latin text. The preliminary Latin text consists of 26 English letters, 2 diacritics and the boundary symbol "-". S12: The initial Latin text is split into components using the boundary symbol. The text is divided into a list of component sequences using a pre-set Manchu component dictionary. The component splitting includes using the boundary symbol "-" as a component segmentation indicator, so that each splitting unit corresponds to the smallest pronunciation unit of a Manchu phoneme. S13: To perform frequency statistics on the component sequence list and correct for biased components, filter out rare components with an occurrence frequency of < 5 to obtain a preliminary component flow; S14: Based on the Manchu phonetic corpus, the component stream is subjected to pronunciation matching and correction, and continuous repeating units are removed to obtain the corrected component stream. The pronunciation matching and correction includes correcting spelling deviations to correct pronunciation units. S15: Clean the corrected component stream to obtain a cleaned component stream.

3. The Manchu speech synthesis method based on character component modeling according to claim 1, characterized in that, The S2 process, based on the cleaned component stream, forms a character embedding sequence. Feature templates are constructed according to internal rules, and the character embedding sequence and feature templates are concatenated in chronological order to obtain a high-dimensional model input tensor. Forming the character embedding sequence includes assigning a global character-ID to each unique character and mapping the ID to a 64-dimensional vector, including: S21: Input the cleaned component stream into a unique character partitioning unit to form a structured character sequence set. The partitioning unit includes lexical segmentation of each character and preservation of its adjacent logical relationships. S22: Based on the character-ID allocation mechanism, a unique character-ID and its index identifier are established for each character in the structured character sequence set, and mapped to a 64-dimensional real vector. The character-ID allocation mechanism has cross-corpus consistency and can be mapped to the subsequent vector space. The 64-dimensional real vector includes weight parameters for representing character semantic information and pronunciation features. S23: Organize the 64-dimensional real vector into a set of Manchu Latin units, which includes core units covering commonly used Manchu pronunciation components and has virtual character slots and real embedded units. S24: Perform consistency verification and quality assessment on the set of Manchu Latin units to obtain the character embedding sequence; S25: Based on the acoustic characteristics of Manchu pronunciation, the character-ID, the corresponding 64-dimensional real vector, and the Manchu-specific acoustic feature dimension containing the fundamental frequency and pitch band are concatenated into a feature template according to the feature dimension. The feature template uses component-level character sequences as the basic unit. S26: Combine the component stream with the feature template, and concatenate the character-ID, character embedding sequence and feature template according to the feature dimension to form a high-dimensional model input tensor.

4. The Manchu speech synthesis method based on character component modeling according to claim 1, characterized in that, S3 feeds the high-dimensional model input tensor into a pre-trained duration prediction sub-network to obtain the predicted phoneme duration. The pre-trained duration prediction sub-network is trained based on a set of feature units and includes a stepwise discriminator that adapts the mapping relationship between one Manchu component and 1-3 Latin transliteration characters. It is optimized using a dual loss function and is trained separately as an independent module in the later stages of the overall model training. S31: Obtain the Manchu text library, first transcribe the Manchu text into Latin, and then clean it to obtain the cleaned text; S32: Using boundary symbols to identify and use a Manchu component dictionary, the cleaned text is split into components at the component level to obtain a component set. The Manchu component dictionary includes 66 component-Latin mappings. S33: Perform frequency statistics on the component set, delete low-frequency components that appear less than 5 times, then perform pronunciation matching and correct deviations based on the speech corpus, remove duplicates from the verified components, and obtain training data containing training units; S34: Treat each training unit in the training data as a component-level character sequence, assign a unique index to each character appearing in the sequence, and map the index to a 64-dimensional vector to form a character embedding matrix; S35: Calculate the logarithmic duration of each training unit in the speech corpus, and construct prosodic features based on the hidden representation generated by the encoder; S36: Construct a duration prediction subnetwork and input prosodic features into it. Through adversarial training, make the duration prediction distribution conform to the duration characteristics of real Manchu speech. The adversarial training includes using a GAN framework, with GAN-DP as an independent module. Its discriminator is a stepwise discriminator that can independently evaluate the variable duration of each Latin transcribing character. During training, a double loss function is used to complete gradient backpropagation. In the later stage of the overall model training, only the duration prediction subnetwork is trained separately, and the weights of the duration prediction subnetwork are iteratively updated. After training, the complete duration prediction subnetwork and character embedding matrix are saved to permanent storage for inference.

5. The Manchu speech synthesis method based on character component modeling according to claim 1, characterized in that, S4 treats each training unit in the training data as a component-level character sequence, assigns a unique index to each character appearing in the sequence, and maps the index to a 64-dimensional vector to form a character embedding matrix, including: S41: Construct a prosodic mask using a pre-trained Manchu-specific prosodic dictionary. This mask identifies the positions of stressed syllables, liaison boundary characters, and word-final stressed syllables on the time axis of the character embedding sequence, while also marking the positions of minor characters in unstressed syllables. S42: The attention weight of minor characters of unstressed syllables in the character embedding sequence is suppressed by the attention weighting unit, and the suppression is implemented in the corresponding attention weighting unit to reduce its interference with prosodic feature extraction; S43: The weighted character embedding sequence is input into the feature aggregation module, and the feature is aggregated through linear mapping and normalization operations to obtain the prosodic feature vector.

6. The Manchu speech synthesis method based on character component modeling according to claim 1, characterized in that, The S5 inputs the character embedding sequence, prosodic mask, and predicted phoneme duration into a pre-trained feature extraction network to obtain a prosodic feature space. This pre-trained feature extraction network is trained based on a set of Manchu-Latin transcribing components, employing a convolutional-Transformer dynamic collaborative hybrid structure. It includes a hybrid structure of small Transformer blocks with residual connections and 3-5 character local convolutional blocks. The outputs of both are fused through dynamic weights that adaptively adjust with the syllable length and stress position of the sequence, including: S51: Obtain the Manchu text library, first transcribe the Manchu text into Latin, and then clean it to obtain the cleaned text; S52: After cleaning, the text is subjected to character filtering, component-level splitting, low-frequency component removal, pronunciation matching correction and deduplication, and then labeled to obtain labeled training data; S53: Based on labeled training data, a Manchu-specific prosodic dictionary is trained; S54: Based on the Manchu-specific prosodic dictionary, a dynamic attention mask is constructed for the character sequences in the labeled training data, and a convolutional block is designed to extract local feature maps. A small Transformer block with residual connections is introduced to perform self-attention. The outputs of the two are fused by dynamic weights that are adaptively adjusted according to the length of the syllables and the stress position of the sequence to obtain a comprehensive prosodic feature representation. S55: Persist the convolutional blocks, Transformer blocks, dynamic attention masks and their weights together to form a pre-trained feature extraction network.

7. The Manchu speech synthesis method based on character component modeling according to claim 1, characterized in that, The S7 is used to load a lightweight multi-band iSTFT decoder for Manchu, performs three-band division and parallel iSTFT processing on the Mel spectrum sequence, combines prosodic feature vectors and optimal alignment paths, and splices waveforms through a Manchu-specific fixed synthesis filter to output a complete synthesized speech waveform, including: S71: The Manchu lightweight multi-band inverse short-time Fourier transform decoder first divides the Mel spectrum sequence into three dedicated frequency bands based on the spectral distribution of Manchu vowels, unvoiced consonants, and voiceless consonants. The division includes a low-frequency band, a mid-frequency band, and a high-frequency band. The spectral distribution of Manchu vowels is 200-800Hz, the spectral distribution of unvoiced consonants is 800-3000Hz, and the spectral distribution of voiceless consonants is 3000-5000Hz. S72: The three dedicated frequency bands are each configured with an independent inverse short-time Fourier transform module, which generates corresponding time-domain waveform segments in parallel; S73: Input the corresponding time-domain waveform segment, and splice it through the frequency band fusion module based on prosodic feature vector and optimal alignment path to obtain a complete synthesized speech waveform. The frequency band fusion module includes a fixed synthesis filter. The Manchu lightweight multi-band inverse short-time Fourier transform decoder uses iSTFT to replace the redundant convolution operation of HiFi-GAN.

8. A Manchu speech synthesis device based on character component modeling, wherein the Manchu speech synthesis device based on character component modeling is used to implement the Manchu speech synthesis method based on character component modeling as described in any one of claims 1-7, characterized in that, The device includes: The data preprocessing module is used to convert the original Manchu text according to the transcription rules, then perform component-level splitting based on a preset Manchu component dictionary, and then clean it to obtain a cleaned component stream. The transcription rules include transcribing the original Manchu text according to the rules in the Manchu-specific Latin character table. The transcription process retains 26 English letters, 2 diacritics, and the boundary symbol "-". The cleaning includes frequency statistics, correction of biased components, and removal of consecutive repeating units. Feature template module: Based on the cleaned component stream, it forms a character embedding sequence, constructs a feature template according to internal rules, and concatenates the character embedding sequence and feature template in chronological order to obtain a high-dimensional model input tensor. The formation of the character embedding sequence includes assigning a global character-ID to each unique character and mapping the ID to a 64-dimensional vector. The construction of the feature template according to internal rules includes concatenating the character-ID, the corresponding 64-dimensional real vector, and the Manchu-specific acoustic feature dimension containing the fundamental frequency and pitch band according to the feature dimension to form a feature template. The Manchu-specific acoustic feature dimension containing the fundamental frequency and pitch band is set according to the acoustic characteristics of Manchu pronunciation. Duration prediction module: It is used to feed the input tensor of the high-dimensional model into the pre-trained duration prediction sub-network to obtain the predicted phoneme duration. The pre-trained duration prediction sub-network is trained based on the feature unit set and contains a stepwise discriminator that adapts the mapping relationship between 1 Manchu component and 1-3 Latin transliteration characters. It is optimized with a double loss function and is trained separately as an independent module in the later stage of the overall model training. Prosodic feature module: used to extract features from the input tensor of the high-dimensional model based on the predicted phoneme duration, and generate prosodic feature vector. The feature extraction includes first constructing a prosodic mask using a Manchu-specific prosodic dictionary, and then using the prosodic mask to suppress minor characters of unstressed syllables, enhance the attention weights of connected speech boundary characters and word-final stressed syllables. Feature extraction module: used to input character embedding sequence, prosodic mask and predicted phoneme duration into pre-trained feature extraction network to obtain prosodic feature space. The pre-trained feature extraction network is trained based on Manchu Latin transcribing component set, adopts convolution-Transformer dynamic collaborative hybrid structure, and includes a hybrid structure of small Transformer blocks with residual connections and 3-5 character local convolution blocks. The outputs of both are fused by dynamic weights that are adaptively adjusted according to the syllable length and stress position of the sequence. Alignment Path Module: Used to perform alignment path search in prosodic feature space using a monotonic alignment search algorithm improved with Gaussian noise, and output the optimal alignment path and the corresponding Mel spectrum sequence; The speech synthesis module loads a lightweight multi-band iSTFT decoder for Manchu, performs three-band division and parallel iSTFT processing on the Mel spectrum sequence, combines prosodic feature vectors and the optimal alignment path, and splices waveforms through a Manchu-specific fixed synthesis filter to output a complete synthesized speech waveform. The passband center frequency of the Manchu-specific fixed synthesis filter is set according to the spectral distribution of Manchu vowels, unvoiced consonants, and voiceless consonants. The low-frequency passband corresponds to the Manchu vowel spectrum distribution of 200-800Hz, the mid-frequency passband corresponds to the unvoiced consonant spectrum distribution of 800-3000Hz, and the high-frequency passband corresponds to the voiceless consonant spectrum distribution of 3000-5000Hz.

9. A Manchu speech synthesis device based on character component modeling, characterized in that, The Manchu speech synthesis device based on character component modeling includes: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images