Reading-guided low-resource speech token speech synthesis method and device

Abstract

The present application relates to a reading guide low-resource speech token speech synthesis method and device, belonging to the technical field of speech synthesis, low-resource language processing, speech token modeling and artificial intelligence generation. It includes: obtaining a speech token level difficult area mask, reliability weight and training action intensity according to the segment level reading method prompt mapping, and performing reliability weighting on the speech token loss; performing pronunciation knowledge internalization training with the reading guide path as the teacher path and without the reading guide path as the student path; inputting the text to be synthesized in the reasoning stage, predicting the speech token through the unguided path, and outputting the synthesized speech through the acoustic generation module and the speech output module. The present application can alleviate the misreading problem of numbers, dates, proper nouns, loanwords, mixed scripts and rare expressions in low-resource speech synthesis, and improve the text consistency, pronunciation correctness and unguided reasoning availability.

Description

Technical Field

[0001] This invention relates to a reading-guided low-resource voice token speech synthesis method and apparatus, belonging to the fields of speech synthesis, low-resource language processing, voice token modeling, text normalization, and artificial intelligence generation technology. Background Technology

[0002] In recent years, speech synthesis technology based on discrete speech tokens has gradually become an important technical route in the field of speech generation. This type of method typically first uses a neural speech coding model to discretize the target speech, mapping the continuous speech waveform into a sequence of discrete speech tokens. Then, a sequence generation model predicts the corresponding speech tokens based on the input text and acoustic conditions. Finally, the target speech is generated through an acoustic generation module and a waveform recovery module. Compared with traditional speech synthesis schemes based on spectral features, the discrete speech token-based method can better unify the modeling process between text sequences and speech sequences, and therefore has been widely applied in scenarios such as multilingual speech synthesis, cross-lingual speech generation, and low-resource language speech generation. However, in low-resource language scenarios, due to the limited scale of high-quality text-to-speech pairing data, the training corpus often fails to fully cover complex pronunciation phenomena such as numbers, dates, times, proper nouns, foreign words, abbreviations, mixed script expressions, and rare expressions. Especially in low-resource or medium-low-resource languages ​​such as Thai and Vietnamese, the same text segment may have multiple legal pronunciations, making it difficult for the model to stably learn a consistent text-to-pronunciation mapping relationship.

[0003] While existing voice token models can generate relatively natural speech, they may still encounter problems in complex pronunciation scenarios, such as sentence-level inconsistencies, unstable digit normalization, errors in switching between mixed languages, and misreading, omissions, or substitutions of difficult-to-read segments, thus affecting the overall speech generation quality. Currently, some existing technologies attempt to reduce the impact of text surface variations through text normalization, rule dictionaries, phoneme conversion, or front-end phonetic annotation. However, these solutions typically focus on rule replacement or single annotation of local text segments, making it difficult to simultaneously constrain both sentence-level reading strategies and local segment-level pronunciation ambiguities. Furthermore, automatically generated pronunciation prompts may contain boundary recognition errors or low-reliability prompts. If unreliable prompts are directly used for model training, incorrect pronunciations may propagate to the voice token model, further affecting overall pronunciation stability.

[0004] Therefore, a novel low-resource speech token-based speech synthesis method is needed that can provide stable reading strategy constraints for the whole sentence during the training phase and provide fine-grained reading prompts for local difficult-to-read segments; at the same time, it can adaptively adjust the training intensity according to the reliability of the prompts, thereby improving the pronunciation stability and speech generation quality of the model in low-resource language scenarios. Summary of the Invention

[0005] The technical problem this invention aims to solve is: addressing the issues in existing low-resource voice token speech synthesis methods, such as insufficient granularity of reading constraints, poor pronunciation stability of complex text segments, high noise in automatic reading prompts, reliance on external front-ends during the inference stage, and the susceptibility to misreading of segments containing numbers, dates, proper nouns, foreign words, and mixed scripts. This invention provides a reading-guided low-resource voice token speech synthesis method and apparatus to improve the pronunciation control capability and speech generation quality of voice token models in low-resource language scenarios.

[0006] The technical solution of the present invention is as follows: Firstly, the present invention provides a reading-guided low-resource speech token-based speech synthesis method, the method comprising:

[0007] Step 1: Obtain training samples for low-resource language speech synthesis and preprocess the training sample text.

[0008] Step 2: Construct a structured reading guidance record; First, generate sentence-level reading constraints based on the training sample text; then generate segment-level reading prompts; finally, construct corresponding verification evidence to form a structured reading guidance record;

[0009] Step 3: Construct the reading guidance prefix; First, serialize the structured reading guidance record into a guidance token sequence; then encode the guidance token sequence using a guidance encoder; finally, generate the reading guidance prefix representation;

[0010] Step 4: Train the voice token generation model; First, concatenate the reading guidance prefix with the text embedding; then input it into the voice token generation model; finally, predict the corresponding discrete voice token sequence.

[0011] Step 5: Perform reliability-weighted training. First, generate speech token-level hard region masks, reliability weights, and training action strengths based on segment-level pronunciation cues. Then, apply reliability weighting to the speech token loss. Finally, enhance the supervision strength of high-confidence pronunciation sensitive regions.

[0012] Step 6: Conduct pronunciation knowledge internalization training; first, construct teacher-generated paths with reading guidance and student-generated paths without reading guidance; then, constrain the output distribution of teacher and student paths in pronunciation-sensitive areas; finally, transfer pronunciation control ability under reading guidance conditions to unguided reasoning paths;

[0013] Step 7: Perform inference generation; first, input the text to be synthesized; then predict the speech token sequence through the unread guided path; finally, generate the target speech through the acoustic generation module and the speech output module.

[0014] Further, Step 1 includes:

[0015] Step 1.1, Training Sample Acquisition: Obtain speech synthesis training data corresponding to low-resource languages;

[0016] Step 1.2, Text Preprocessing: The training sample text is uniformly encoded to eliminate character encoding differences between different data sources; punctuation is normalized to retain pause-related punctuation and merge duplicate punctuation; hybrid script detection is performed to identify the locations where target language characters, Latin letters, numbers, symbols, or foreign language fragments coexist.

[0017] Step 1.3, Hard-to-read segment labeling: Label the local text regions that may lead to incorrect speech token predictions with hard-to-read segments to form the final training samples.

[0018] Furthermore, Step 2 includes:

[0019] Step 2.1: Generate sentence-level reading constraints; the sentence-level reading constraints shall include at least the text normalization tendency, sentence pause position, phrase division method, dominant language, script consistency strategy, mixed language reading strategy, and cross-language pronunciation migration risk;

[0020] Step 2.2: Generate fragment-level reading prompts; the fragment-level reading prompts include at least the surface text fragment, fragment start and end positions, fragment type, local reading prompts, normalized text reading, target language character reading, phoneme sequence, syllable boundaries, natural language pronunciation explanation information, training actions, and reliability scores;

[0021] Step 2.3: Generate verification evidence; the verification evidence includes structural legality verification, fragment boundary alignment verification, normalized reading consistency verification, script consistency verification, ambiguity risk verification, and hallucination risk verification;

[0022] Step 2.4: Construct a structured reading guidance record;

[0023] The fragment types include one or more of the following: numbers, dates, times, amounts, proper nouns, abbreviations, foreign words, mixed script text, rare expressions, and language-specific variant expressions;

[0024] When the reliability score is lower than a preset threshold, the corresponding segment is subjected to training actions such as downweighting, discarding, or retaining only sentence-level constraints.

[0025] Furthermore, the specific processing flow of Step 3 includes:

[0026] For the input text sequence:

[0027]

[0028] wherein n represents the length of the input text, and xi represents the i-th text unit; the structured reading guidance record is processed by a serialization function to obtain a guidance token sequence:

[0029]

[0030] wherein G(X) represents the structured reading guidance record corresponding to the input text X; Φ(·) represents the guidance record serialization function; the guidance token sequence is input into a guidance encoder to obtain a reading guidance prefix representation:

[0031]

[0032] wherein GuideEnc(·) represents the guidance encoder; Hg represents the reading guidance prefix; the text sequence is processed by a text embedding layer to obtain a text representation:

[0033]

[0034] wherein Embed(·) represents the text embedding function; the reading guidance prefix and the text representation are concatenated to obtain a conditional input representation:

[0035]

[0036] wherein [·;·] represents the vector concatenation operation; the conditional input representation is input into a speech token generation model to obtain the conditional probability of the t-th speech token:

[0037]

[0038] wherein yt represents the t-th speech token; y<t represents the first t-1 speech tokens; Fθ represents the speech token generation model; Ht represents the hidden state at the t-th step.

[0039] Further, the guidance encoder respectively performs embedding representation on the hierarchy type, segment type, segment number, segment position, pronunciation action and reliability score in the structured reading guidance record;

[0040] and generates the reading guidance prefix through concatenation, weighted summation, gating fusion or attention fusion.

[0041] Further, the specific processing procedure of Step 5 includes:

[0042] First, a speech token level difficulty region mask is constructed according to the segment level pronunciation prompt:

[0043]

[0044] Where Mt=1 indicates that the t-th voice token is located in the pronunciation-sensitive region; training action intensity coefficients are generated based on the training actions:

[0045]

[0046] Where at represents the training action; Ψ(·) represents the training action mapping function; reliability weights are generated based on the reliability score:

[0047]

[0048] Where rt represents the reliability score corresponding to the t-th voice token; τ represents the reliability threshold; and the weighted supervision coefficient of the t-th voice token is expressed as:

[0049]

[0050] Where λ1 represents the enhancement coefficient for the difficult region; the prediction loss for the t-th voice token is expressed as:

[0051]

[0052] The overall reliability weighted loss function is expressed as:

[0053]

[0054] Where T represents the length of the voice token sequence;

[0055] The voice token-level hard region mask is obtained by coarsely mapping the text segment position to the voice token time region;

[0056] The mapping is only used for loss weighting and is not used as a mandatory duration alignment target;

[0057] The intensity of the training action is determined based on whether the segment requires enhanced supervision, maintained supervision, reduced supervision, or discarded supervision.

[0058] Furthermore, the specific processing flow of Step 6 includes:

[0059] For the same input text, construct paths with and without reading guidance respectively;

[0060] The probability distribution of the next voice token, including the reading guidance path, is represented as follows:

[0061]

[0062] The probability distribution of the next voice token without a reading guidance path is represented as follows:

[0063]

[0064] Knowledge internalization training is performed on the pronunciation-sensitive areas, and the internalization loss function is expressed as follows:

[0065]

[0066] Where KL(·||·) represents the KL divergence function; StopGrad(·) represents the stop gradient operation; PwT(yt) and PnT(yt) represent the probability distributions after temperature scaling, respectively;

[0067] The final model training objective function is expressed as:

[0068]

[0069] Where λ2 represents the weight coefficient of knowledge internalization loss.

[0070] Furthermore, the reasoning phase does not rely on external reading guidance records by default;

[0071] When the text to be synthesized contains high-risk, difficult-to-read segments, enable the enhanced guided path;

[0072] It outputs synthesized speech with or without guidance, depending on the user's settings.

[0073] Furthermore, the acoustic generation module includes one or more of an acoustic decoder, a streaming acoustic model, a diffusion acoustic model, or a conditional acoustic decoder;

[0074] The speech output module includes one or more of a neural vocoder, a parametric vocoder, or an end-to-end waveform generator.

[0075] Secondly, the present invention provides a reading-guided low-resource voice token speech synthesis device, the device comprising: a module for executing the reading-guided low-resource voice token speech synthesis method.

[0076] The beneficial effects of this invention are:

[0077] 1. This invention improves pronunciation stability in low-resource language with complex reading scenarios by constructing a structured reading guidance record, providing a unified reading strategy at the sentence level and fine-grained reading prompts at the segment level;

[0078] 2. By introducing verification evidence and reliability score mechanisms, this invention can reduce the interference of incorrect reading prompts on the training process of voice token models and reduce the risk of low-reliability prompts being amplified by the model.

[0079] 3. This invention strengthens the training intensity of pronunciation-sensitive regions such as numbers, dates, proper nouns, foreign words, and mixed scripts by using hard region masks and reliability-weighted loss functions, thereby reducing the misreading rate of complex text fragments;

[0080] 4. This invention improves the stability of speech generation when the model does not rely on external guidance during the inference stage by using a pronunciation knowledge internalization training mechanism to enable the model to learn pronunciation control ability with a reading guidance path without a reading guidance path.

[0081] 5. This invention can effectively improve the pronunciation accuracy, robustness to complex text, and actual deployment availability in the inference stage of low-resource language speech token speech synthesis tasks. Attached Figure Description

[0082] Figure 1 This is a flowchart of the reading-guided low-resource voice token speech synthesis method of the present invention;

[0083] Figure 2 This is a flowchart of the reading-guided low-resource voice token speech synthesis method of the present invention. Detailed Implementation

[0084] Example 1: As Figures 1-2 As shown, in a first aspect, the present invention provides a reading-guided low-resource speech token speech synthesis method, the method comprising:

[0085] Step 1: Obtain training samples for low-resource language speech synthesis and preprocess the training sample text.

[0086] Step 2: Construct a structured reading guidance record; First, generate sentence-level reading constraints based on the training sample text; then generate segment-level reading prompts; finally, construct corresponding verification evidence to form a structured reading guidance record;

[0087] Step 3: Construct the reading guidance prefix; First, serialize the structured reading guidance record into a guidance token sequence; then encode the guidance token sequence using a guidance encoder; finally, generate the reading guidance prefix representation;

[0088] Step 4: Train the voice token generation model; First, concatenate the reading guidance prefix with the text embedding; then input it into the voice token generation model; finally, predict the corresponding discrete voice token sequence.

[0089] Step 5: Perform reliability-weighted training. First, generate speech token-level hard region masks, reliability weights, and training action strengths based on segment-level pronunciation cues. Then, apply reliability weighting to the speech token loss. Finally, enhance the supervision strength of high-confidence pronunciation sensitive regions.

[0090] Step 6: Conduct pronunciation knowledge internalization training; first, construct teacher-generated paths with reading guidance and student-generated paths without reading guidance; then, constrain the output distribution of teacher and student paths in pronunciation-sensitive areas; finally, transfer pronunciation control ability under reading guidance conditions to unguided reasoning paths;

[0091] Step 7: Perform inference generation; first, input the text to be synthesized; then predict the speech token sequence through the unread guided path; finally, generate the target speech through the acoustic generation module and the speech output module.

[0092] Further, Step 1 includes:

[0093] Step 1.1, Training Sample Acquisition: Obtain speech synthesis training data corresponding to low-resource languages;

[0094] Step 1.2, Text Preprocessing: The training sample text is uniformly encoded to eliminate character encoding differences between different data sources; punctuation is normalized to retain pause-related punctuation and merge duplicate punctuation; hybrid script detection is performed to identify the locations where target language characters, Latin letters, numbers, symbols, or foreign language fragments coexist.

[0095] Step 1.3, Hard-to-read segment labeling: Label the local text regions that may lead to incorrect speech token predictions with hard-to-read segments to form the final training samples.

[0096] Furthermore, Step 2 includes:

[0097] Step 2.1: Generate sentence-level reading constraints; the sentence-level reading constraints shall include at least the text normalization tendency, sentence pause position, phrase division method, dominant language, script consistency strategy, mixed language reading strategy, and cross-language pronunciation migration risk;

[0098] Step 2.2: Generate fragment-level reading prompts; the fragment-level reading prompts include at least the surface text fragment, fragment start and end positions, fragment type, local reading prompts, normalized text reading, target language character reading, phoneme sequence, syllable boundaries, natural language pronunciation explanation information, training actions, and reliability scores;

[0099] Step 2.3: Generate verification evidence; the verification evidence includes structural legality verification, fragment boundary alignment verification, normalized reading consistency verification, script consistency verification, ambiguity risk verification, and hallucination risk verification;

[0100] Step 2.4: Construct a structured reading guidance record;

[0101] The fragment type includes one or more of numbers, dates, time, amounts, proper nouns, abbreviations, foreign words, mixed-script texts, rare expressions, and language-specific variant pronunciation expressions;

[0102] When the reliability score is lower than a preset threshold, a training action of weight reduction, discarding, or retaining only sentence-level constraints is performed on the corresponding fragment.

[0103] Further, the specific processing flow of Step3 includes:

[0104] For the input text sequence:

[0105]

[0106] Wherein, n represents the length of the input text, and xi represents the i-th text unit; the structured reading guidance record is processed by a serialization function to obtain a guidance token sequence:

[0107]

[0108] Wherein, G(X) represents the structured reading guidance record corresponding to the input text X; Φ(·) represents the guidance record serialization function; the guidance token sequence is input into a guidance encoder to obtain a reading guidance prefix representation:

[0109]

[0110] Wherein, GuideEnc(·) represents the guidance encoder; Hg represents the reading guidance prefix; the text sequence passes through the text embedding layer to obtain a text representation:

[0111]

[0112] Wherein, Embed(·) represents the text embedding function; the reading guidance prefix and the text representation are concatenated to obtain a conditional input representation:

[0113]

[0114] Wherein, [·;·] represents the vector concatenation operation; the conditional input representation is input into a speech token generation model to obtain the conditional probability of the t-th speech token:

[0115]

[0116] Wherein, yt represents the t-th speech token; y<t represents the first t-1 speech tokens; Fθ represents the speech token generation model; Ht represents the hidden state at the t-th step.

[0117] Furthermore, the guidance encoder embeds the hierarchical type, segment type, segment number, segment position, reading action, and reliability score in the structured reading guidance record.

[0118] Reading guidance prefixes are generated through splicing, weighted summation, gating fusion, or attention fusion.

[0119] Furthermore, the specific processing flow of Step 5 includes:

[0120] First, construct a voice token-level hard region mask based on fragment-level reading cues:

[0121]

[0122] Where Mt=1 indicates that the t-th voice token is located in the pronunciation-sensitive region; training action intensity coefficients are generated based on the training actions:

[0123]

[0124] Where at represents the training action; Ψ(·) represents the training action mapping function; reliability weights are generated based on the reliability score:

[0125]

[0126] Where rt represents the reliability score corresponding to the t-th voice token; τ represents the reliability threshold; and the weighted supervision coefficient of the t-th voice token is expressed as:

[0127]

[0128] Where λ1 represents the enhancement coefficient for the difficult region; the prediction loss for the t-th voice token is expressed as:

[0129]

[0130] The overall reliability weighted loss function is expressed as:

[0131]

[0132] Where T represents the length of the voice token sequence;

[0133] The voice token-level hard region mask is obtained by coarsely mapping the text segment position to the voice token time region;

[0134] The mapping is only used for loss weighting and is not used as a mandatory duration alignment target;

[0135] The intensity of the training action is determined based on whether the segment requires enhanced supervision, maintained supervision, reduced supervision, or discarded supervision.

[0136] Furthermore, the specific processing flow of Step 6 includes:

[0137] For the same input text, construct paths with and without reading guidance respectively;

[0138] The probability distribution of the next voice token, including the reading guidance path, is represented as follows:

[0139]

[0140] The probability distribution of the next voice token without a reading guidance path is represented as follows:

[0141]

[0142] Knowledge internalization training is performed on the pronunciation-sensitive areas, and the internalization loss function is expressed as follows:

[0143]

[0144] Where KL(·||·) represents the KL divergence function; StopGrad(·) represents the stop gradient operation; PwT(yt) and PnT(yt) represent the probability distributions after temperature scaling, respectively;

[0145] The final model training objective function is expressed as:

[0146]

[0147] Where λ2 represents the weight coefficient of knowledge internalization loss.

[0148] Furthermore, the reasoning phase does not rely on external reading guidance records by default;

[0149] When the text to be synthesized contains high-risk, difficult-to-read segments, enable the enhanced guided path;

[0150] It outputs synthesized speech with or without guidance, depending on the user's settings.

[0151] Furthermore, the acoustic generation module includes one or more of an acoustic decoder, a streaming acoustic model, a diffusion acoustic model, or a conditional acoustic decoder;

[0152] The speech output module includes one or more of a neural vocoder, a parametric vocoder, or an end-to-end waveform generator.

[0153] Example 2: As Figures 1-2 As shown, in a first aspect, the present invention provides a reading-guided low-resource speech token speech synthesis method, the method comprising:

[0154] Step 1: Obtain training samples for low-resource language speech synthesis and preprocess the training sample text; wherein, the low-resource language includes at least one of Thai or Vietnamese; the preprocessing includes unified encoding, punctuation normalization, mixed script detection, and hard-to-read segment marking;

[0155] The system acquires low-resource language text, corresponding speech, speaker identifiers, language identifiers, and optional quality labels; performs unified encoding, punctuation normalization, mixed script detection, and hard-to-read segment recognition on the text; and performs resampling, silence trimming, volume normalization, and speech tokenization on the speech to finally obtain text with hard-to-read segment tags and corresponding speech token sequences.

[0156] Further, step 1 includes:

[0157] Step 1.1, Training Sample Construction: First, collect low-resource language text and corresponding speech data; the low-resource languages ​​include Thai, Vietnamese or other languages ​​with complex pronunciation phenomena such as numbers, dates, loanwords, mixed scripts and proper nouns; each training sample includes the original text, speech waveform, language identifier, speaker identifier and sample number;

[0158] Step 1.2, Text Preprocessing: Perform unified character encoding on texts from different sources; standardize punctuation marks, retain punctuation marks related to pauses and merge repeated symbols; perform mixed script detection to identify the positions where target language characters, numbers, Latin letters and symbols appear together; further identify candidate difficult-to-read segments corresponding to numbers, dates, times, amounts, proper nouns, foreign words, abbreviations and rare expressions;

[0159] Step 1.3, Difficult-to-read segment identification: Candidate difficult-to-read segments are identified using one or more of the following methods: rule matching, vocabulary coverage detection, language model perplexity analysis, speech recognition back-translation comparison, or manual annotation; for numbers, dates, and time expressions, their original text form and candidate standardized pronunciations are preserved; for proper nouns and foreign words, their possible ambiguous pronunciation states are recorded.

[0160] Step 1.4, Speech-side processing: Resampling, silence trimming, and volume normalization are performed on the speech samples; then, a speech token sequence is obtained through a discrete speech representation model, a neural audio encoder, or a speech token extraction model; if a sample has speech that is too short, segmented incorrectly, has abnormal volume, or fails to be tokenized, it is marked as a low-quality sample.

[0161] Step 2: Construct a structured reading guidance record; First, generate sentence-level reading constraints based on the training sample text; then generate segment-level reading prompts; finally, construct corresponding verification evidence to form a structured reading guidance record;

[0162] Furthermore, a structured pronunciation guidance record is constructed for the input text; the structured pronunciation guidance record includes sentence-level pronunciation constraints, fragment-level pronunciation prompts, and verification evidence; wherein the sentence-level pronunciation constraints are used to describe the pronunciation strategy at the whole sentence level; the fragment-level pronunciation prompts are used to describe the local pronunciation of candidate difficult-to-read fragments; and the verification evidence is used to determine the reliability of the pronunciation prompts;

[0163] Further, step 2 includes:

[0164] Step 2.1, Sentence-level pronunciation constraint construction: Generate sentence-level pronunciation strategies based on text content; the sentence-level pronunciation strategies include text normalization tendency, intra-sentence pause positions, phrase segmentation methods, dominant language, script consistency strategy, and mixed language pronunciation strategy;

[0165] Step 2.2, Segment-level pronunciation cue construction: For each candidate difficult segment, record its surface text, start and end positions, segment type, local pronunciation cue, standardized text pronunciation, target language script pronunciation, phoneme sequence, syllable boundaries, and natural language pronunciation explanation information;

[0166] Step 2.3, Verification of Evidence Construction: Perform structural legality verification, fragment boundary alignment verification, normalized pronunciation consistency verification, script consistency verification, and ambiguity risk verification on the automatically generated pronunciation prompts; when the verification results do not meet the credibility conditions, set the corresponding fragment to reduced supervision or discard supervision;

[0167] Step 2.4, Structured Record Serialization: Convert the structured pronunciation guidance record into a guide token sequence; retain sentence-level fields, fragment fields, verification fields, and fragment order information during the serialization process.

[0168] Step 3: Construct the reading guidance prefix; First, serialize the structured reading guidance record into a guidance token sequence; then encode the guidance token sequence using a guidance encoder; finally, generate the reading guidance prefix representation;

[0169] Furthermore, the structured pronunciation guidance record is encoded as a pronunciation guidance prefix; then the pronunciation guidance prefix and text embedding are input into the speech token generation model, so that the model receives both text information and structured pronunciation information during the training phase.

[0170] Further, step 3 includes:

[0171] Step 3.1, Pronunciation guidance encoding: The guidance token sequence is encoded through an embedding layer, a lightweight Transformer, a gated fusion network, or an attention fusion network to obtain the pronunciation guidance prefix representation;

[0172] Step 3.2, Conditional Control Access: Concatenate pronunciation guidance prefixes with text embeddings, perform attention interactions, or fuse intra-layer conditional biases; enable the model to distinguish between sentence-level pronunciation constraints, fragment-level pronunciation cues, and verification evidence;

[0173] Step 3.3, Voice Token Prediction: The voice token generation model performs autoregressive voice token prediction based on text context and pronunciation guidance information; the final output is a predicted voice token sequence.

[0174] Furthermore, the specific processing flow of step 3 includes:

[0175] For the input text sequence:

[0176]

[0177] Constructing a structured pronunciation guide record:

[0178]

[0179] The bootstrap token sequence is then obtained through a serialization function:

[0180]

[0181] in, This represents the serialization function. This represents the sequence of guide tokens corresponding to the structured pronunciation guide record;

[0182] Input the guide token sequence into the pronunciation guide encoder:

[0183]

[0184] in, This indicates a pronunciation guide encoder. Indicates a pronunciation guide prefix;

[0185] Text embedding is represented as:

[0186]

[0187] The pronunciation guide prefix is ​​then concatenated with the text embedding to obtain the model input state:

[0188]

[0189] in, This represents a sequence concatenation operation;

[0190] The model performs autoregressive voice token prediction based on the input state and outputs a predicted voice token sequence:

[0191]

[0192] in, This represents the predicted sequence of voice tokens.

[0193] Step 4: Train the voice token generation model; First, concatenate the reading guidance prefix with the text embedding; then input it into the voice token generation model; finally, predict the corresponding discrete voice token sequence.

[0194] Furthermore, a reliability score is calculated for each difficult segment based on the verification evidence; the difficult segments are mapped to the corresponding speech token regions; and then training supervision for high-reliability difficult regions is enhanced through reliability-weighted loss.

[0195] Further, step 4 includes:

[0196] Step 4.1, Reliability Scoring: Calculate the segment reliability score based on structural legality, boundary alignment, normalized reading consistency, and script consistency; reduce the reliability score for segments with high ambiguity risk or hallucination risk;

[0197] Step 4.2, Hard Region Mapping: Based on the position of the segment in the text and the length of the voice token, roughly estimate the corresponding voice token region; the mapping is only used for training loss weighting;

[0198] Step 4.3, Reliability-weighted training: Increase training weights for high-reliability, difficult regions; reduce or discard supervision for low-reliability segments.

[0199] Step 4.4, Training Action Control: Training actions include strengthening supervision, maintaining supervision, reducing supervision, and discarding supervision; higher confidence and higher risk segments adopt stronger supervision strategies, while lower confidence segments adopt weak supervision or unsupervised strategies.

[0200] Furthermore, the specific processing flow of step 4 includes:

[0201] For the A fragment Calculate the reliability score based on the verification evidence:

[0202]

[0203] in, Represents the Sigmoid function; Indicates the score for structural legality; Indicates the boundary alignment score; This indicates the score for consistency in normalized readings; Indicates the script consistency score; Indicates the ambiguity risk score; Indicates the risk score of hallucination;

[0204] Then, the voice token region marker is obtained based on the difficult region mapping:

[0205]

[0206] in, Indicates the first Does the voice token belong to a difficult area? The area representing the voice token corresponding to the segment;

[0207] Further training weights are obtained:

[0208]

[0209] in, Indicates the weighting coefficient for the difficult region; Indicates the intensity of training movements; Indicates the reliability threshold;

[0210] The model training loss is expressed as:

[0211]

[0212] in, Indicates the first Predicted loss for each voice token.

[0213] Step 5: Perform reliability-weighted training; First, generate speech token-level hard region masks, reliability weights, and training action strengths based on segment-level pronunciation cues; then, apply reliability weighting to the speech token loss; finally, enhance the supervision strength of high-confidence pronunciation sensitive regions.

[0214] Furthermore, during the training phase, both pronunciation-guided paths and pronunciation-free paths are established simultaneously; the pronunciation-guided path serves as the teacher's path, while the pronunciation-free path serves as the student's path; through consistent learning, the student learns pronunciation control ability from the pronunciation-guided path.

[0215] Further, step 5 includes:

[0216] Step 5.1, Guided Path Training: Input pronunciation guidance prefixes and text embeddings into the model to generate a speech token distribution with stable pronunciation constraints;

[0217] Step 5.2, Unguided Path Training: Input only text embeddings to enable the model to learn speech generation capabilities without external pronunciation cues;

[0218] Step 5.3, Internalization of pronunciation knowledge: Through distributional consistency constraints, unguided paths learn pronunciation control knowledge from guided paths;

[0219] Step 5.4, Joint Loss Optimization: The basic speech token prediction loss, reliability-weighted loss, and pronunciation knowledge internalization loss are used together for model training.

[0220] Furthermore, the specific processing flow of step 5 includes:

[0221] Output the probability distribution of the next voice token for both guided and unguided paths:

[0222]

[0223] in, This indicates that there is a guided path for output distribution; This indicates the output distribution without a guide path;

[0224] Then, the internalization loss of pronunciation knowledge was calculated:

[0225]

[0226] in, Represents the divergence measure; This indicates that the gradient operation has been stopped.

[0227] The final total training loss is:

[0228]

[0229] in, This indicates the predicted loss based on the basic voice token. Indicates the reliability-weighted loss weight; This represents the weight of the internalization loss of pronunciation knowledge.

[0230] Step 6: Perform pronunciation knowledge internalization training; first, construct a teacher-generated path with reading guidance and a student-generated path without reading guidance; then, constrain the output distribution of the teacher path and the student path in the pronunciation sensitive area; finally, transfer the pronunciation control ability under the reading guidance condition to the unguided reasoning path;

[0231] Step 7: Perform inference generation; first, input the text to be synthesized; then predict the speech token sequence through a no-reading guided path; finally, generate the target speech through the acoustic generation module and the speech output module.

[0232] The inference stage uses a silent guided path by default, and only the text to be synthesized is input to predict the speech token; then the final synthesized speech is output through the acoustic generation module and vocoder.

[0233] Further, step 7 includes:

[0234] Step 7.1, Default Unguided Inference: No structured pronunciation guidance record is generated; only voice token prediction is performed based on the input text.

[0235] Step 7.2, Enhanced Guided Reasoning: When the input text contains high-risk, difficult-to-read segments, construct a structured pronunciation guidance record and enable enhanced pronunciation control;

[0236] Step 7.3, Abnormal rollback handling: When the structured pronunciation guidance record is missing, verification fails, or reliability is insufficient, the system automatically rolls back to the unguided reasoning path;

[0237] Step 7.4, Speech Output: Input the predicted speech token into the acoustic synthesis module and vocoder, and finally output the synthesized speech.

[0238] Secondly, the present invention provides a reading-guided low-resource voice token speech synthesis device, the device comprising: a module for executing the reading-guided low-resource voice token speech synthesis method.

[0239] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the reading-guided low-resource voice token speech synthesis method.

[0240] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the reading-guided low-resource voice token speech synthesis method.

[0241] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the reading-guided low-resource voice token speech synthesis method.

[0242] Experimental verification and technical effect analysis.

[0243] This embodiment is used to verify the technical effectiveness of the above-mentioned technical solution. The experimental comparison objects include methods such as F5-TTS, MMS-TTS, CosyVoice2, CosyVoice3, and ReGuide-TTS. Evaluation metrics include CER, WER, SS, UTMOS, Nat. MOS, and Pron. MOS, where lower CER and WER indicate better text consistency, and higher Pron. MOS indicates better pronunciation accuracy.

[0244] Table 1 Comparison of objective indicators between Thai and Vietnamese

[0245] Thai F5-TTS 5.74 9.48 71.6 4.12 Thai MMS-TTS 5.51 9.07 68.4 4.01 Thai CosyVoice2 4.92 8.41 72.8 4.20 Thai CosyVoice3 5.06 8.67 74.7 4.32 Thai ReGuide-TTS 3.92 7.18 74.2 4.30 Vietnamese F5-TTS 6.76 10.95 70.6 4.04 Vietnamese MMS-TTS 6.63 10.72 67.8 3.94 Vietnamese CosyVoice2 5.93 9.62 72.2 4.15 Vietnamese CosyVoice3 6.05 9.78 73.9 4.26 Vietnamese ReGuide-TTS 4.97 8.62 73.6 4.24

[0246] Table 2 Comparison of Subjective Indicators between Thai and Vietnamese

[0247] Thai F5-TTS 4.14±0.08 4.01±0.10 Thai MMS-TTS 3.98±0.11 4.03±0.11 Thai CosyVoice2 4.17±0.08 4.18±0.07 Thai CosyVoice3 4.30±0.07 4.14±0.09 Thai ReGuide-TTS 4.28±0.08 4.40±0.09 Vietnamese F5-TTS 4.06±0.14 3.84±0.20 Vietnamese MMS-TTS 3.92±0.17 3.90±0.21 Vietnamese CosyVoice2 4.13±0.12 4.03±0.18 Vietnamese CosyVoice3 4.22±0.10 4.00±0.17 Vietnamese ReGuide-TTS 4.21±0.13 4.12±0.17

[0248] As shown in Tables 1 and 2, ReGuide-TTS reduced CER and WER in both Thai and Vietnamese, and achieved better results in subjective pronunciation scores, indicating that structured reading guidance can improve text consistency and pronunciation accuracy in low-resource speech synthesis.

[0249] Table 3 Comparison of error rates for difficult passages at different reading guidance granularities

[0250] Qwen2.5-0.5B ST-LM 5.46 9.41 8.18 8.83 9.67 10.49 Flat Reading Guide 4.92 8.38 7.16 8.05 8.54 9.32 Utterance-level Guide 4.32 7.96 7.32 7.74 7.92 8.86 Span-level Guide 4.81 7.08 5.58 6.66 7.39 8.04 Hierarchical Guide 4.09 6.32 5.66 6.13 6.78 7.36

[0251] Table 3 shows that hierarchical guidance is superior to single sentence-level or single segment-level guidance. Sentence-level constraints are more conducive to the stability of overall reading strategies, while segment-level prompts are more beneficial for locally difficult segments. Combining the two can achieve a more balanced reduction in difficulty across categories such as numbers, time, proper nouns, foreign words, and rare expressions. Table 4 shows the results of reliability learning and knowledge internalization ablation.

[0252] Full 5.04 8.46 5.78 6.24 0.46 w / o Verification 5.71 9.12 6.52 7.08 0.56 w / o Reliability Calib. 5.31 8.67 6.18 6.86 0.68 w / o Hard Weighted CE 5.66 9.20 6.28 7.22 0.94 w / o Internalization 5.42 8.93 5.93 7.48 1.55 w / o Hierarchical Guide 5.83 9.43 6.84 7.43 0.59

[0253] Table 4 shows that removing verification, reliability calibration, weighting of difficult regions, or knowledge internalization all lead to performance degradation; among them, the gap between Guide CER and No-guide CER is significantly widened after removing knowledge internalization, indicating that pronunciation knowledge internalization can narrow the gap between guided and unguided paths.

[0254] In summary, the technical effects of the method described above are consistent with the beneficial effects described in the invention: structured reading guidance improves reading control, reliability learning reduces the impact of noise cues, and internalization of pronunciation knowledge enhances unguided reasoning ability. In the experiments, CER and WER were used to measure the consistency between the synthesized speech and the input text, SS was used to measure speaker similarity or relevance similarity, UTMOS and Nat. MOS were used to measure naturalness, and Pron. MOS was used to measure pronunciation accuracy. These metrics collectively correspond to the beneficial effects of this invention, such as text consistency, pronunciation accuracy, and deployment usability.

[0255] The experimental results show that ReGuide-TTS does not only improve a single subjective metric, but also achieves a consistent improvement across both objective error rate and subjective pronunciation evaluation. This indicates that structured reading guidance, reliability learning, and knowledge internalization work together to enhance the low-resource speech token synthesis process.

[0256] The advantages of hierarchical guidance show that simple whole-sentence prompts are difficult to cover local difficult-to-read segments, while simple segment prompts are difficult to maintain the consistency of whole-sentence reading strategies. By combining sentence-level constraints and segment-level prompts, the model can obtain both global and local reading control information at the same time.

[0257] In the ablation experiment, removing validation or reliability calibration reduced performance, indicating that automatic reading prompts cannot be used indiscriminately as strong supervision; removing weighting of difficult regions weakened learning of pronunciation-sensitive segments; and removing knowledge internalization widened the gap between guided and unguided paths.

[0258] Therefore, the experimental verification of the present invention has a direct correspondence with Steps 2 to 6 in Example 1, which can prove that structured recording, reliability weighting and knowledge internalization all contribute to the final technical effect.

[0259] Figure 1 The training phase corresponds to Steps 1 to 6 of Example 1, and the inference phase corresponds to Step 7. The upper part of the figure shows how structured reading guidance participates in training, and the lower part shows the model generating speech under the default no-guidance path and the optional enhanced guidance path.

[0260] Figure 1 This also highlights a key boundary of the invention: reading guidance primarily serves as verifiable supervision and optional inference enhancement during training, without requiring the construction of an external reading record for each inference phase. This boundary ensures the deployability of the method.

[0261] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A reading-guided low-resource speech token-based speech synthesis method, characterized in that: The method includes: Step 1: Obtain training samples for low-resource language speech synthesis and preprocess the training sample text. Step 2: Construct a structured reading guidance record; First, generate sentence-level reading constraints based on the training sample text; then generate segment-level reading prompts; finally, construct corresponding verification evidence to form a structured reading guidance record; Step 3: Construct the reading guidance prefix; First, serialize the structured reading guidance record into a guidance token sequence; then encode the guidance token sequence using a guidance encoder; finally, generate the reading guidance prefix representation; Step 4: Train the voice token generation model; First, concatenate the reading guidance prefix with the text embedding; then input it into the voice token generation model; finally, predict the corresponding discrete voice token sequence. Step 5: Perform reliability-weighted training. First, generate speech token-level hard region masks, reliability weights, and training action strengths based on segment-level pronunciation cues. Then, apply reliability weighting to the speech token loss. Finally, enhance the supervision strength of high-confidence pronunciation sensitive regions. Step 6: Conduct pronunciation knowledge internalization training; first, construct teacher-generated paths with reading guidance and student-generated paths without reading guidance; then, constrain the output distribution of teacher and student paths in pronunciation-sensitive areas; finally, transfer pronunciation control ability under reading guidance conditions to unguided reasoning paths; Step 7: Perform inference generation; first, input the text to be synthesized; then predict the speech token sequence through the unread guided path; finally, generate the target speech through the acoustic generation module and the speech output module.

2. The reading-guided low-resource speech token speech synthesis method according to claim 1, characterized in that: Step 1 includes: Step 1.1, Training Sample Acquisition: Obtain speech synthesis training data corresponding to low-resource languages; Step 1.2, Text Preprocessing: The training sample text is uniformly encoded to eliminate character encoding differences between different data sources; punctuation is normalized to retain pause-related punctuation and merge duplicate punctuation; hybrid script detection is performed to identify the locations where target language characters, Latin letters, numbers, symbols, or foreign language fragments coexist. Step 1.3, Hard-to-read segment labeling: Label the local text regions that may lead to incorrect speech token predictions with hard-to-read segments to form the final training samples.

3. The reading-guided low-resource speech token speech synthesis method according to claim 1, characterized in that: Step 2 includes: Step 2.1: Generate sentence-level reading constraints; the sentence-level reading constraints shall include at least the text normalization tendency, sentence pause position, phrase division method, dominant language, script consistency strategy, mixed language reading strategy, and cross-language pronunciation migration risk; Step 2.2: Generate fragment-level reading prompts; the fragment-level reading prompts include at least the surface text fragment, fragment start and end positions, fragment type, local reading prompts, normalized text reading, target language character reading, phoneme sequence, syllable boundaries, natural language pronunciation explanation information, training actions, and reliability scores; Step 2.3: Generate verification evidence; the verification evidence includes structural legitimacy verification, segment boundary alignment verification, normalized pronunciation consistency verification, script consistency verification, ambiguity risk verification, and hallucination risk verification; Step 2.4: Construct a structured reading guidance record; The segment type includes one or more of numbers, dates, time, amounts, proper nouns, abbreviations, loanwords, mixed script texts, rare expressions, and language-specific variant pronunciation expressions; When the reliability score is lower than a preset threshold, a training action of weight reduction, discarding, or retaining only sentence-level constraints is performed on the corresponding segment.

4. The reading-guided low-resource speech token speech synthesis method according to claim 1, characterized in that: The specific processing flow of Step 3 includes: For the input text sequence: ； where n represents the length of the input text, and xi represents the i-th text unit; the structured reading guidance record is processed by a serialization function to obtain a guidance token sequence: ； where G(X) represents the structured reading guidance record corresponding to the input text X; Φ(·) represents the guidance record serialization function; the guidance token sequence is input into a guidance encoder to obtain a reading guidance prefix representation: ； where GuideEnc(·) represents the guidance encoder; Hg represents the reading guidance prefix; the text sequence passes through a text embedding layer to obtain a text representation: ； where Embed(·) represents the text embedding function; the reading guidance prefix and the text representation are concatenated to obtain a conditional input representation: ； where [·;·] represents the vector concatenation operation; the conditional input representation is input into a speech token generation model to obtain the conditional probability of the t-th speech token: ； where yt represents the t-th speech token; y<t represents the first t-1 speech tokens; Fθ represents the speech token generation model; Ht represents the hidden state at the t-th step.

5. A reading-guided low-resource speech token speech synthesis method according to Claim 1, characterized in that: The guidance encoder respectively performs embedding representation on the hierarchy type, segment type, segment number, segment position, pronunciation action and reliability score in the structured reading guidance record; and generates the reading guidance prefix by means of concatenation, weighted summation, gating fusion or attention fusion.

6. The reading-guided low-resource speech token speech synthesis method according to claim 1, characterized in that: The specific processing flow of Step 5 includes: First, construct a speech token-level difficult region mask according to the segment-level pronunciation prompt: ； where Mt=1 indicates that the t-th speech token is located in a pronunciation-sensitive region; generate a training action intensity coefficient according to the training action: ； where at represents the training action; Ψ(·) represents the training action mapping function; generate a reliability weight according to the reliability score: ； where rt represents the reliability score corresponding to the t-th speech token; τ represents the reliability threshold; the weighted supervision coefficient of the t-th speech token is expressed as: ； where λ1 represents the difficult region enhancement coefficient; the prediction loss of the t-th speech token is expressed as: ； The overall reliability weighted loss function is expressed as: ； where T represents the length of the speech token sequence; The speech token-level difficult region mask is obtained by coarsely mapping the text segment position to the speech token time region; The mapping is only used for loss weighting, and is not used as a mandatory duration alignment target; The intensity of the training action is determined based on whether the segment requires enhanced supervision, maintained supervision, reduced supervision, or discarded supervision.

7. The reading-guided low-resource speech token speech synthesis method according to claim 1, characterized in that: The specific processing steps in Step 6 include: For the same input text, construct paths with and without reading guidance respectively; The probability distribution of the next voice token, including the reading guidance path, is represented as follows: ； The probability distribution of the next voice token without a reading guidance path is represented as follows: ； Knowledge internalization training is performed on the pronunciation-sensitive areas, and the internalization loss function is expressed as follows: ； Where KL(·||·) represents the KL divergence function; StopGrad(·) represents the stop gradient operation; PwT(yt) and PnT(yt) represent the probability distributions after temperature scaling, respectively; The final model training objective function is expressed as: ； Where λ2 represents the weight coefficient of knowledge internalization loss.

8. The reading-guided low-resource speech token speech synthesis method according to claim 1, characterized in that: The reasoning phase does not rely on external reading guidance records by default; When the text to be synthesized contains high-risk, difficult-to-read segments, enable the enhanced guided path; It outputs synthesized speech with or without guidance, depending on the user's settings.

9. A reading-guided low-resource speech token-based speech synthesis method according to claim 1, characterized in that: The acoustic generation module includes one or more of the following: an acoustic decoder, a streaming acoustic model, a diffusion acoustic model, or a conditional acoustic decoder. The speech output module includes one or more of a neural vocoder, a parametric vocoder, or an end-to-end waveform generator.

10. A reading-guided low-resource voice token speech synthesis device, characterized in that, The apparatus includes a module for performing the reading-guided low-resource voice token speech synthesis method as described in any one of claims 1 to 5.

Images