Speech synthesis model optimization method and device, computer equipment and storage medium

Abstract

The invention provides a speech synthesis model optimization method and device, computer equipment and a storage medium, and relates to the technical field of speech synthesis. According to the method, the original text is input into the zero-sample speech synthesis model to generate the first speech, the original text and the first speech are input into the multi-layer speech evaluation model, quality evaluation is performed on the first speech, and the error time period is accurately recognized. And based on the voice evaluation result, generating the mask text corresponding to the error time period, thereby realizing accurate positioning and marking of the error part. And re-inputting the mask text into the zero sample speech synthesis model to generate the second speech, and performing quality evaluation on the second speech, thereby providing a clear optimization direction for adjustment of model parameters. The model parameters are iteratively updated through the direct preference optimization algorithm, so that the optimization efficiency and accuracy of the zero-sample speech synthesis model are improved, and the performance and reliability of the zero-sample speech synthesis model in practical applications such as financial intelligent customer service and auxiliary medical service are improved.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a method, apparatus, computer device and storage medium for optimizing a speech synthesis model. Background Technology

[0002] With the rapid development of artificial intelligence technology, zero-shot text-to-speech (TTS) technology has gradually become a research hotspot in the field of speech synthesis. This technology can generate natural speech similar to the target speaker with only a small number of speech samples, and is widely used in virtual dubbing, podcast generation, and digital human voice interaction. In the financial and healthcare industries, the application prospects of zero-shot TTS technology are particularly broad. For example, in financial services, it can be used for voice interaction in intelligent customer service to improve user experience; in the medical field, it can be used to assist medical equipment with voice prompts and patient communication, improving the convenience and efficiency of medical services.

[0003] Currently, the mainstream technical solutions for achieving zero-shot TTS are mainly based on autoregressive (AR) models and non-autoregressive (NAR) models. Autoregressive models generate speech frame by frame, which performs well in terms of naturalness, but when generating long texts, error accumulation can easily lead to speech interruptions, repetitions, or abnormal pauses, affecting speech coherence and information integrity in professional scenarios. While non-autoregressive models significantly improve synthesis efficiency and support parallel generation, they generally suffer from problems such as speech drift, semantic mismatch, and local pronunciation distortion, making it difficult to meet the stringent requirements for speech quality and consistency in fields such as finance and healthcare.

[0004] Existing zero-shot TTS systems still face the following technical challenges: Insufficient speech stability leads to frequent misreadings and interruptions. In the autoregressive model, errors in the early stages of sequence generation propagate and amplify throughout the generation process, resulting in a decline in subsequent speech quality and phenomena such as word misreading and abnormal pauses. Although the non-autoregressive model avoids error propagation, it is prone to problems such as unclear pronunciation and rhythmic instability due to the lack of temporal dependency modeling, especially in the synthesis of technical terms and long sentences. Sound quality and semantic consistency are difficult to guarantee. Synthesized speech often contains acoustic flaws such as background noise, abnormal formants, and abrupt pitch changes, which seriously affect the auditory experience and information intelligibility. Existing systems generally lack the ability to automatically detect and correct "local abnormal segments." Once a pronunciation error or quality degradation occurs, the entire speech usually needs to be resynthesized, which not only incurs high computational costs and response delays but also fails to meet the real-time requirements of applications such as finance and healthcare. There is a lack of fine-grained speech quality assessment mechanisms. Current systems mostly rely on external automatic speech recognition (ASR) models or signal quality indicators for evaluation. These methods have a single evaluation dimension, are limited by the performance of external models, and cannot accurately locate the error type and location in the time dimension, such as pronunciation errors of specific words, abnormal silence segments, and inaccurate intonation, making it impossible to implement targeted corrections. The user preference alignment mechanism is imperfect. Although some works have attempted to introduce techniques such as human feedback reinforcement learning (RLHF) or direct preference optimization (DPO) to improve the naturalness of speech, the existing feedback signals are mostly sentence-level overall scores, lacking preference annotations at the granular and temporal alignment level of speech segments. Therefore, it is difficult to guide the model to perform fine-grained optimization of local pronunciation features, which restricts the further improvement of speech personalization and scene adaptability.

[0005] Existing zero-shot TTS models still have significant shortcomings in terms of stability, fidelity, and fine-grained evaluation feedback mechanisms, especially in fields such as finance and healthcare where high speech quality and accuracy are required. Therefore, improving the accuracy of zero-shot speech synthesis model optimization to achieve high-quality, high-accuracy, and natural and fluent speech synthesis output has become an urgent technical problem to be solved. Summary of the Invention

[0006] This application provides a method, apparatus, device, and storage medium for optimizing a speech synthesis model, aiming to solve the technical problem that existing zero-shot speech synthesis models have low optimization accuracy and poor model optimization results in application fields such as finance and medicine where the requirements for speech quality and accuracy are extremely high. This is due to the lack of stability, fidelity, and fine-grained evaluation feedback mechanisms.

[0007] In a first aspect, this application provides a method for optimizing a speech synthesis model, the method comprising the following steps: The original text is input into the zero-shot speech synthesis model to generate the first speech; The original text and the first speech are input into a multi-layer speech evaluation model. The first speech is then evaluated for quality using the multi-layer speech evaluation model to generate a speech evaluation result. Based on the speech evaluation results, a mask text corresponding to the erroneous time period in the first speech is generated; The masked text is input into the zero-shot speech synthesis model to generate a second speech; The quality of the second speech is evaluated using the multi-layer speech evaluation model to obtain the audio score of the corresponding segment of the second speech. Using the audio segment score as an immediate reward signal, the model parameters of the zero-shot speech synthesis model are iteratively updated through a direct preference optimization algorithm until the speech generated by the zero-shot speech synthesis model meets a preset quality threshold.

[0008] Secondly, this application also provides an optimization device for a speech synthesis model, the optimization device for the speech synthesis model comprising: The first speech generation module is used to input the original text into the zero-shot speech synthesis model to generate the first speech; The first speech evaluation module is used to input the original text and the first speech into a multi-layer speech evaluation model, perform quality evaluation on the first speech through the multi-layer speech evaluation model, and generate a speech evaluation result. The mask text generation module is used to generate mask text corresponding to the error period in the first speech based on the speech evaluation result; The second speech generation module is used to input the masked text into the zero-sample speech synthesis model to generate the second speech; The second speech evaluation module is used to evaluate the quality of the second speech through the multi-layer speech evaluation model and obtain the segment audio score corresponding to the second speech. The model parameter optimization module is used to use the audio segment score as an instant reward signal and iteratively update the model parameters of the zero-shot speech synthesis model through a direct preference optimization algorithm until the speech generated by the zero-shot speech synthesis model meets a preset quality threshold.

[0009] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the optimization method for the speech synthesis model as described above.

[0010] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the optimization method for the speech synthesis model as described above.

[0011] This application provides a method, apparatus, computer device, and storage medium for optimizing a speech synthesis model. The method generates first speech by inputting the original text into a zero-shot speech synthesis model, providing a foundational speech sample for subsequent quality assessment and optimization. The original text and the first speech are then input into a multi-layer speech evaluation model to assess the quality of the first speech, generating detailed evaluation results. These results not only provide quantitative indicators of speech quality but also accurately identify erroneous segments in the speech through multi-dimensional and fine-grained evaluation. Based on the speech evaluation results, masked text corresponding to the erroneous segments in the first speech is generated, achieving precise location and labeling of the erroneous parts, improving optimization accuracy, avoiding the high computational cost of traditional overall regeneration, and enhancing optimization efficiency. The masked text is then re-inputted into the zero-shot speech synthesis model to generate second speech, further improving the efficiency and quality of speech synthesis. The second speech is then evaluated using a multi-layer speech evaluation model to obtain segment audio scores, directly reflecting the optimized speech quality and providing clear feedback for adjusting model parameters, ensuring the correctness of the optimization direction. Using audio segment scores as immediate reward signals, the model parameters are iteratively updated through a direct preference optimization algorithm until the generated speech meets a preset quality threshold. By maximizing the reward signal, the zero-shot speech synthesis model is ensured to adjust its parameters in each iteration in the direction of improving speech quality, which significantly improves the efficiency and accuracy of zero-shot speech synthesis model optimization. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for an optimization method of a speech synthesis model according to an embodiment of the present invention; Figure 2 A flowchart illustrating an embodiment of a speech synthesis model optimization method provided in this application; Figure 3 A schematic diagram of the data transmission structure of an embodiment of a speech synthesis model optimization method provided in this application; Figure 4 This is a schematic diagram of the structure of an embodiment of a speech synthesis model optimization device provided in this application. Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0014] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the described order. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0017] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The optimization method for the speech synthesis model provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. When the server receives the raw text sent by the client, it can input the raw text into a zero-shot speech synthesis model to generate first speech; input the raw text and the first speech into a multi-layer speech evaluation model, and perform a quality evaluation on the first speech to generate a speech evaluation result; based on the speech evaluation result, generate mask text corresponding to the error periods in the first speech; input the mask text into the zero-shot speech synthesis model to generate second speech; perform a quality evaluation on the second speech through the multi-layer speech evaluation model to obtain a segment audio score corresponding to the second speech; use the segment audio score as an immediate reward signal, and iteratively update the model parameters of the zero-shot speech synthesis model through a direct preference optimization algorithm until the speech generated by the zero-shot speech synthesis model meets a preset quality threshold.

[0020] In this invention, existing zero-shot speech synthesis models suffer from low optimization accuracy and poor optimization results in applications such as finance and healthcare, where high speech quality and accuracy are crucial. This is due to shortcomings in stability, fidelity, and fine-grained evaluation feedback mechanisms. The invention addresses this by inputting the original text into the zero-shot speech synthesis model to generate a first speech sample, providing a foundation for subsequent quality assessment and optimization. The original text and the first speech sample are then input into a multi-layer speech evaluation model to perform a quality assessment on the first speech, generating detailed evaluation results. This not only provides quantitative indicators of speech quality but also accurately identifies erroneous segments in the speech through multi-dimensional, fine-grained evaluation. Based on the speech evaluation results, masked text corresponding to the erroneous segments in the first speech is generated, enabling precise location and labeling of the errors, improving optimization accuracy, avoiding the high computational cost of traditional overall regeneration, and enhancing optimization efficiency. Finally, the masked text is re-inputted into the zero-shot speech synthesis model to generate a second speech sample, further improving both the efficiency and quality of speech synthesis. The quality of the second speech is evaluated using a multi-layer speech evaluation model to obtain segment audio scores, which directly reflect the optimized speech quality and provide clear feedback for adjusting model parameters, ensuring the correctness of the optimization direction. Using the segment audio scores as an immediate reward signal, the model parameters are iteratively updated through a direct preference optimization algorithm until the generated speech meets a preset quality threshold. By maximizing the reward signal, the zero-shot speech synthesis model ensures that its parameters are adjusted in each iteration to improve speech quality, significantly enhancing the efficiency and accuracy of zero-shot speech synthesis model optimization.

[0021] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0022] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of a speech synthesis model optimization method provided in this application.

[0023] like Figure 2 As shown, the optimization method of the speech synthesis model includes steps S101 to S106.

[0024] S101. Input the original text into the zero-shot speech synthesis model to generate the first speech; In one embodiment, the server receives the original text from the client or the upstream application. The original text can be natural language text such as sentences or paragraphs of any length, including news reports, novel fragments, conversation content, etc., for example, "Welcome to use the intelligent zero-shot speech synthesis model".

[0025] Before inputting the original text into the zero-shot speech synthesis model, it is necessary to preprocess the original text, including text cleaning, text normalization, word segmentation and phoneticization processing, etc. Specifically, text cleaning is used to remove redundant spaces, special characters or non-standard symbols in the original text; text normalization is used to convert numbers, abbreviations, special symbols, etc. in the original text into corresponding full spellings or standard readings, for example, converting "2025" to "two thousand and twenty-five"; word segmentation is used to segment the original text into words or phrases; phoneticization is used to convert the segmented words or phrases into a sequence of phonemes as an input form that is easier for the zero-shot speech synthesis model to process.

[0026] Load the pre-trained zero-shot speech synthesis (TextToSpeech, TTS) model from the storage device to ensure that the model parameters are correctly loaded and in a runnable state. The zero-shot speech synthesis model has the ability to learn the speaker's characteristics based on the reference speech and synthesize new speech, and can generate natural speech similar to the target speaker with only a small number of speech samples. The core of the zero-shot speech synthesis model is to learn the speaker's characteristics through a small amount of speech data, so as to imitate the speech style, tone color and intonation of the speaker when generating speech.

[0027] The zero-shot speech synthesis model is usually based on deep learning technology, especially neural network architectures such as autoregressive models, non-autoregressive models or their variants. Among them, the generation process of the autoregressive model (such as the Tacotron series) is frame-by-frame, that is, the next frame is predicted based on the already generated speech frames, with high generation quality but slow speed; the generation process of the non-autoregressive model (such as the FastSpeech series) is parallel, and all speech frames can be generated at once, with fast generation speed but possible sacrifice of some naturalness.

[0028] Input the preprocessed original text into the zero-shot speech synthesis model. The zero-shot speech synthesis model internally uses a pre-trained language model (such as BERT, GPT, etc.) or a phoneme encoder to map each phoneme in the text to a corresponding sequence of tokens (the smallest processing unit) to obtain a phoneme sequence; input the phoneme sequence into the acoustic model to generate corresponding acoustic features (such as Mel spectrogram); then input the generated acoustic features into the vocoder, and the vocoder converts the acoustic features into a playable speech waveform signal, that is, the first speech.

[0029] This embodiment preprocesses the original text to ensure the standardization and accuracy of the input text. The zero-shot speech synthesis model learns the speaker's features using a small number of speech samples, and can generate natural speech similar to the target speaker. This not only improves the efficiency and quality of speech synthesis, but also enhances the adaptability and flexibility of the zero-shot speech synthesis model, enabling it to be widely used in various scenarios and providing users with a more natural and fluent voice interaction experience.

[0030] S102. Input the original text and the first speech into the multi-layer speech evaluation model, and perform quality evaluation on the first speech through the multi-layer speech evaluation model to generate speech evaluation results. A multi-layer speech evaluation model is a complex system for assessing the quality of generated speech. Its core function is to quantitatively evaluate the quality of generated speech through multi-layered analysis and comparison. The model consists of several sub-modules, including a speech feature encoder, a text semantic encoder, a timestamp predictor, and an evaluation network. The speech feature encoder uses a self-supervised learning model (such as wav2vec2.0) to extract semantic features from the speech signal; the text semantic encoder uses a pre-trained language model (such as BERT) to convert text into a phoneme-level token sequence; the timestamp predictor uses a Transformer-based model to predict the temporal position of each phoneme or word in the speech signal; and the evaluation network assesses the semantic consistency between speech and text, outputting a speech quality score and error time periods. These sub-modules work together to evaluate key indicators such as speech stability, sound quality, and semantic consistency from different perspectives.

[0031] In one embodiment, the first speech output by the zero-shot speech synthesis model and the preprocessed original text are simultaneously input into a multi-layer speech evaluation model. The multi-layer speech evaluation model performs multi-dimensional and fine-grained alignment and comparison of the original text and the first speech, and comprehensively evaluates and locates errors in the speech from multiple levels (such as acoustic quality, semantic consistency, and temporal alignment), thereby accurately evaluating the generation quality of the first speech output by the zero-shot speech synthesis model and locating specific problems that may exist therein.

[0032] Specifically, the first speech signal undergoes preprocessing operations such as framing and windowing to adapt to the input requirements of the speech feature encoder. The preprocessed speech signal is then input into the speech feature encoder to extract semantic features from the first speech signal. The input raw text undergoes preprocessing operations such as word segmentation and phoneme conversion to generate a phoneme-level token sequence. This phoneme sequence is input into the text semantic encoder to generate a semantic feature representation of the text. A Transformer-based timestamp prediction model is used to predict the start and end timestamps of each phoneme or word in the speech signal. The predicted timestamps are calibrated to ensure accuracy and consistency. The speech features and text semantic features are input into an evaluation network for phoneme-speech semantic alignment. The network outputs a global audio quality score, text matching error, and error time periods.

[0033] Further, the original text and the first speech are input into a multi-layer speech evaluation model to generate speech semantic content features and phoneme sequences; the speech semantic content features and the phoneme sequences are input into a sequence evaluation network in the multi-layer speech evaluation model to perform phoneme speech semantic alignment and semantic extraction, and output the speech evaluation result.

[0034] In one embodiment, the original text and the first speech are simultaneously input into a multi-layer speech evaluation model, and feature extraction operations are performed on the original text and the first speech respectively through the multi-layer speech evaluation model.

[0035] The original text is converted into a phoneme sequence. Specifically, the original text is cleaned and normalized, and a predefined phoneme dictionary is used to map each word or character in the text to its corresponding phoneme, thus obtaining a phoneme sequence.

[0036] The speech semantic content features that reflect the deep semantic content are extracted from the first speech. Specifically, the first speech is resampled and normalized to ensure that its format and sampling rate meet the requirements of the model input. A pre-trained self-supervised model is used to extract features from the preprocessed first speech, and the output features are dimensionality reduced or mapped through a linear layer or a small network to obtain speech semantic content features that match the dimension of the phoneme sequence.

[0037] The multi-layer speech evaluation model includes a speech feature encoder and a text semantic encoder.

[0038] The main function of a text semantic encoder is to convert raw text into a discrete, structured sequence of phonemes. A phoneme is the smallest unit of speech in a language, and converting raw text into a phoneme sequence helps the model to understand pronunciation rules more accurately and align them with the speech signal.

[0039] The main function of the speech feature encoder is to extract a sequence of feature vectors from the original audio waveform of the first speech that can reflect its deep semantic content. Different from traditional acoustic features (such as MFCC), the speech semantic content features extracted by this speech feature encoder focus more on the semantic information of the speech rather than pure acoustic characteristics.

[0040] Furthermore, input the original text into the text semantic encoder, and input the first speech into the speech feature encoder; convert the original text into the phoneme sequence through the text semantic encoder; perform feature extraction on the first speech through the speech feature encoder to obtain the speech semantic content features.

[0041] As Figure 3 shown, input the first speech and the original text into a multi-layer speech evaluation model (MLE), and extract features through a speech feature encoder (SpeechEncoder) module and a text semantic encoder (TextEncoder) module respectively.

[0042] Among them, the text semantic encoder (TextEncoder) is used to convert the input original text into a phoneme sequence and extract the original text semantic features. BERT or a similar pre-trained language model can be selected as the text semantic encoder. Preprocess the input original text, including text cleaning and text normalization, tokenize the preprocessed original text, and split the continuous character sequence into meaningful words. According to a predefined phoneme dictionary, map each word or character to its corresponding initial phoneme sequence, for example, the English word "hello" is mapped to [HH,AH,L,OW]; the Chinese character "北" is mapped to [b,ei3] (where 3 represents the tone). Convert each phoneme in the initial phoneme sequence into a unique integer ID (TokenID), for example, [HH,AH,L,OW] may be mapped to [101,102,103,104]. Input the TokenID sequence into an embedding layer of the text semantic encoder to convert each discrete ID into a continuous, high-dimensional vector representation. This step converts the symbolized phonemes into a mathematical form that the model can calculate. Output a sequence of high-dimensional vectors, that is, the phoneme sequence, whose length is equal to the number of phonemes in the text, and each vector represents the semantic information of a phoneme.

[0043] The main function of the speech feature encoder is to extract semantic features from the input speech signal. These features can reflect the semantic content and context information of the speech. A pre-trained self-supervised learning model can be selected as the speech feature encoder. Resample the input first speech to uniformly resample the audio to the sampling rate required by the model, and normalize the resampled first speech. Normalize the amplitude of the audio waveform so that its mean is 0 and variance is 1. Input the preprocessed audio waveform into the selected self-supervised learning model. Inside the self-supervised learning model, the preprocessed audio waveform is encoded through multiple convolutional layers and Transformer layers. Extract the hidden states from the middle layer or the last layer of the self-supervised model as features. These hidden states usually have rich semantic information. Since the features output by the self-supervised model may have a high dimension, in order to match the text feature dimension and reduce the computational amount, usually a linear layer (LinearLayer) or a small network is used to reduce the dimension or map the features, and a high-dimensional vector sequence is output, that is, the speech semantic content features.

[0044] The speech semantic content features and the phoneme sequence will be sent into the sequence-to-sequence evaluation network (such as Figure 3 the Evaluator module in it). The Evaluator adopts the sequence-to-sequence Transformer model architecture to align and extract semantics from the speech semantic content features and the phoneme sequence, so as to generate the speech evaluation result.

[0045] Phoneme-speech semantic alignment refers to aligning the semantic features in the speech signal with the text phoneme sequence in time to ensure the semantic consistency between the speech and the text. This process needs to use timestamp information to accurately match the speech features and the phoneme sequence. Semantic extraction refers to extracting key semantic information from the aligned speech and text features to evaluate the quality of the speech.

[0046] Specifically, input the phoneme sequence into the encoder of the sequence-to-sequence evaluation network. The phoneme sequence first undergoes positional encoding to inject the order information of the phoneme sequence. The phoneme sequence passes through multiple Transformer encoder layers. In each layer, each phoneme interacts with all other phonemes in the sequence through the self-attention mechanism to capture the global context information. The encoder finally outputs a sequence of phoneme representations rich in context information. For example, the encoder will learn the close association between "north" and "Beijing" in the word "Beijing".

[0047] The speech semantic content features are also positionally encoded before being input into the decoder. In the first layer of the decoder, a masked self-attention mechanism is used, ensuring that each speech frame only focuses on the frames preceding it. At each layer of the decoder, a cross-attention calculation is performed. The query for this cross-attention calculation is the speech feature representation from the previous layer of the decoder, and the key and value are the text representation output by the encoder, i.e., the phoneme representation sequence. Attention weights are obtained by calculating the similarity between the query and the key. These weights indicate which phonemes should be focused on for each speech frame. For example, for a speech frame pronouncing "b," the attention weight will be highly concentrated on the "b" phoneme in the phoneme sequence. The decoder ultimately outputs a hidden state sequence that integrates speech and text information. Each position in this sequence contains a mixture of information: the content of the current speech frame and the corresponding text content.

[0048] Semantic information for evaluation is extracted from the hidden state sequence, which integrates speech and text information. The hidden state sequence already contains information such as whether the speech matches the text and the quality of the speech. The multi-layer speech evaluation model then "decodes" this information into specific evaluation metrics through subsequent fully connected layers (linear layers).

[0049] The multilayer speech evaluation model extracts three dimensions of speech evaluation results from the hidden state sequence through three independent linear layers (fully connected layers). These speech evaluation results include the overall audio quality score, text matching error, and error time period.

[0050] Specifically, firstly, the hidden state sequence is compressed into a single vector through a global average pooling operation, and then mapped to a scalar score (e.g., 0-5) through a linear layer. This scalar score represents the overall audio quality score, used to evaluate the overall naturalness, clarity, and fluency of the first speech. Secondly, the feature vector of each time step in the hidden state sequence is passed through a linear layer, outputting a scalar. This scalar represents the matching confidence or error probability between the speech content and the corresponding text at that time step, serving as the text matching error. This quantifies the difference between the speech recognition result and the original text, reflecting semantic consistency. Finally, the feature vector of each time step in the hidden state sequence is passed through a linear layer and a sigmoid activation function, outputting a value between 0 and 1, representing the probability of an error at that time step. By setting a threshold (e.g., 0.5), the probability sequence can be converted into a binary sequence (0 for correct, 1 for incorrect). Post-processing (e.g., concatenating adjacent error frames) yields the specific error time period, used to precisely locate the temporal positions of pronunciation errors, repetitions, abnormal pauses, and other problems in the speech.

[0051] This embodiment employs a multi-layered speech evaluation model to comprehensively and finely assess the quality of the first speech generated by a zero-shot speech synthesis model. It not only comprehensively evaluates the speech from multiple dimensions such as acoustic quality, semantic consistency, and temporal alignment, but also accurately locates errors within the speech. Semantic features of the speech and text are extracted using a speech feature encoder and a text semantic encoder, respectively. A sequence-to-sequence evaluation network is then used for phoneme-speech semantic alignment and semantic extraction. The final output includes a detailed evaluation result comprising an overall audio quality score, text matching error, and error time periods. This provides direct feedback signals for optimizing the speech synthesis model and data support for subsequent local corrections and preference optimization, thereby improving the stability and generation quality of the zero-shot speech synthesis model.

[0052] S103. Based on the speech evaluation results, generate mask text corresponding to the error period in the first speech; In one embodiment, the speech evaluation results include a global audio quality score, text matching error, and error time periods. The error time periods specify the exact time intervals in which problems exist in the speech (such as pronunciation errors, semantic inconsistencies, abnormal pauses, etc.), and can be used to identify speech segments that need correction.

[0053] Masked text refers to a modified version of the original text, in which the text portions corresponding to the erroneous time segments are replaced with special mask markers. The purpose of masked text is to instruct the speech synthesis model to regenerate only the speech of these erroneous portions without affecting the speech of the other correct portions.

[0054] Timestamp information can be used to precisely align erroneous time periods with phonemes or words in the original text, identifying which text content corresponds to the incorrect speech segments. The aligned erroneous text is then replaced with a mask marker, leaving the rest unchanged. The masked text preserves contextual information while clearly indicating the parts that need correction.

[0055] Further, based on the error time period, the text segment in the original text corresponding to the error time period is determined; the text segment is marked as a mask region to generate the mask text.

[0056] In one embodiment, the error time period is a time interval extracted from the speech evaluation results, representing the problematic portion of the speech. These time periods need to be mapped to specific text segments in the original text to determine which text content corresponds to the error portion of the speech. The mapping process relies on timestamp information, which precisely aligns phonemes or words in the speech signal with their positions in the text.

[0057] Specifically, error time periods can be extracted from the speech evaluation results. These time periods represent problematic parts of the speech. Error time periods are typically expressed in seconds, such as [0.5, 1.2] and [3.0, 3.5], indicating errors within these time intervals. To map this temporal information to the text, a bridge needs to be established—an alignment relationship between the text and the speech. This alignment relationship records the start and end times of each basic unit (such as a phoneme, word, or phrase) in the original text within the generated speech signal.

[0058] Alignment can be performed using the attention alignment matrix within a zero-shot TTS model or external alignment tools (such as timestamp predictors). For example, the output of a timestamp predictor can be used to precisely align timestamps in speech with phonemes or words in text. The timestamp predictor outputs the start and end times of each phoneme or word, which can be used to determine which text content corresponds to the incorrect time period. The alignment result can be an alignment list where each element contains a text unit and its corresponding time interval.

[0059] Using the alignment list described above, "erroneous time periods" are converted into "erroneous text segments." Specifically, for each erroneous time period in the speech evaluation results, the alignment list is used to find all text units whose time intervals overlap with the current erroneous time period. All text units that overlap with the erroneous time period are merged to form a continuous text segment. For example, if the erroneous time period (0.6s, 0.8s) overlaps with the time interval [0.5s, 0.9s] of the word "Beijing," then the word "Beijing" is identified as the text segment that needs correction.

[0060] Masked regions refer to the text segments in the original text that correspond to the erroneous time periods. These regions are marked as the parts that need correction, usually using special mask markers. The definition of masked regions needs to take into account contextual information to ensure that the speech synthesis model can understand the semantic background during correction.

[0061] Masking strategies such as special token masks, tag-surround masks, and placeholder masks can be used to modify erroneous text fragments in the original text and generate the final masked text. Specifically, a special token mask replaces the erroneous fragment with a special token reserved in the vocabulary; a tag-surround mask adds specific start and end tags before and after the erroneous text fragment; and a placeholder mask replaces the erroneous text with placeholders (such as asterisks ***) that are approximately the same length as the erroneous text.

[0062] Masked text is text generated by replacing masked regions in the original text with mask markers. Masked text preserves the contextual information of the original text while explicitly indicating the parts that need correction. The purpose of generating masked text is to guide the speech synthesis model to regenerate only the erroneous parts, thereby improving the efficiency and accuracy of correction.

[0063] In one embodiment, after determining the masking strategy, the start and end indices of all identified "erroneous text fragments" are found in the original text string. Based on the masking strategy, characters within these index ranges in the original text are replaced with mask markers. For example, if the original text is "I love Beijing Tiananmen", the erroneous fragment is "Beijing", and the masking strategy is special markers, the replacement is "I love [MASK] Tiananmen". If multiple non-contiguous erroneous fragments exist, the above replacement operation is performed on each fragment separately. For example, if the original text is "The weather is great today, suitable for going out", and the erroneous fragments are "weather" and "go out", the generated masked text might be "The weather is great today, suitable for [MASK] to play". Finally, a string containing mask markers is output, i.e., the masked text.

[0064] This embodiment precisely aligns speech and text, mapping erroneous segments in the speech to text fragments and generating masked text using masking markers. This guides the speech synthesis model to regenerate only the erroneous parts. This local correction method not only improves the efficiency of speech synthesis and avoids regenerating the entire speech, but also ensures the naturalness and coherence of the speech by preserving contextual information. Furthermore, flexible masking strategies (such as special markers, label enclosing, or placeholders) further enhance the model's adaptability to different error types, improving the accuracy and semantic consistency of speech synthesis, ultimately significantly optimizing the quality of speech synthesis and the user experience.

[0065] S104. Input the masked text into the zero-sample speech synthesis model to generate the second speech; Masked text is specially processed text in which erroneous parts are replaced with masked markers (such as [MASK]). The purpose of this text is to guide zero-shot speech synthesis (TTS) models to regenerate only the erroneous parts, while preserving the correct parts of the speech. This effectively improves the efficiency and accuracy of speech synthesis, avoiding unnecessary regeneration of the entire speech.

[0066] Specifically, the generated masked text is input into the zero-shot speech synthesis model. The masked text preserves contextual information while explicitly indicating the parts that need correction. Inside the zero-shot speech synthesis model, the scope of generation can be limited by setting an attention mask. Specifically, the model's self-attention mechanism is restricted as follows: for non-masked text tokens, the model can only focus on itself and the generated speech frames; for masked tokens, the model can focus on all text tokens and the generated speech frames, thereby utilizing contextual information to predict the speech in the masked portion.

[0067] Upon receiving masked text, the zero-shot speech synthesis model identifies the mask markers and regenerates only those parts. Based on the masked text, the model generates corrected speech segments. During generation, it utilizes contextual information to produce natural and fluent speech segments, ensuring the corrected speech is semantically consistent with the context. The generated corrected speech segments are then precisely inserted into the corresponding erroneous time periods in the first speech, replacing the original erroneous segments to form the second speech.

[0068] Compared to the first speech, the second speech is identical in content, but the speech quality is significantly improved in the segments covered by the mask markers. Errors (such as pronunciation errors, repetitions, and abnormal pauses) are successfully corrected. Therefore, the second speech is superior to the first speech in both semantic consistency and speech quality, especially in the corrected parts.

[0069] This embodiment achieves precise correction of speech errors by inputting masked text into a zero-shot speech synthesis model. The masked text replaces the erroneous portions with masked markers, guiding the zero-shot speech synthesis model to regenerate only the erroneous parts while preserving the correct parts. The zero-shot speech synthesis model uses contextual information to generate the corrected speech segments and precisely inserts them into the corresponding error time periods in the first speech, replacing the original erroneous segments. This method not only improves the efficiency of speech synthesis and avoids regenerating the entire speech, but also significantly improves speech quality, correcting pronunciation errors, repetitions, and abnormal pauses, making the second speech superior to the first speech in both semantic consistency and naturalness.

[0070] S105. The quality of the second speech is evaluated using the multi-layer speech evaluation model to obtain the segment audio score corresponding to the second speech. Unlike the quality assessment of the first speech segment, the quality assessment of the second speech segment does not score the entire second speech segment, but rather performs a refined assessment of the corrected local speech segments. The multi-layer speech assessment model can focus the assessment attention on specific time intervals based on information from the masked text.

[0071] A segment audio score is a quantitative assessment of the quality of a specific segment (usually a corrected portion) in second speech. The score reflects the overall performance of the corrected speech segment in terms of semantic consistency, pronunciation accuracy, and naturalness. The segment audio score can be a numerical value ranging from 0 to 1, where 1 represents perfect and 0 represents completely unacceptable.

[0072] Further, the second speech is input into the multi-layer speech evaluation model to obtain a local quality score only for the time period corresponding to the masked text; the local quality score is normalized and weighted by confidence to output the audio segment score.

[0073] The local quality score refers to the quality assessment result for a specific time segment in the second speech (i.e., the time segment corresponding to the masked text). This score is specifically used to measure the performance of the corrected speech segment in terms of semantic consistency, pronunciation accuracy, and speech naturalness.

[0074] Specifically, the corrected second speech and the original text are input into a multi-layer speech evaluation model. The second speech is generated by inputting the masked text into a zero-shot speech synthesis model, where errors have been corrected. The multi-layer speech evaluation model evaluates the second speech, paying particular attention to the time segments corresponding to the masked text.

[0075] The multi-layer speech evaluation model first extracts features from the second speech, then the speech feature encoder extracts the semantic features of the second speech, and the text semantic encoder converts the original text into a phoneme sequence. The semantic features of the second speech are then aligned with the phoneme sequence of the original text to ensure the temporal correspondence between the speech and the text.

[0076] Based on the aligned features, a multi-layer speech evaluation model assesses the quality of the second speech segment. Specifically, by utilizing the positional information of the mask markers ([MASK]) in the masked text and combining it with the text-speech alignment relationship, the corresponding target time interval in the second speech segment is mapped back. This target time interval is the "corrected segment" to be evaluated. Within the multi-layer speech evaluation model, an attention masking mechanism can be used when calculating the evaluation score. When calculating cross-attention, a very low weight is applied to speech frames in non-target time intervals, forcing the model to concentrate computational resources on the target time interval.

[0077] From the multi-dimensional output of the multi-layer speech evaluation model, a local quality score specifically for the "corrected segment" is extracted. This local quality score includes the local quality score, text matching error, and error time period. Specifically, since the multi-layer speech evaluation model focuses on the local corrected segment during evaluation, its overall output quality score can be approximated as the local quality score of that corrected segment. The text matching error output is analyzed, and the error value at the text position corresponding to the corrected segment is extracted as the segment-level semantic accuracy score, i.e., the text matching error. The output of the error time period is checked to confirm whether there are still uncorrected errors within the corrected segment. If the error time period is empty or no longer contains the corrected segment, the correction is considered successful.

[0078] Based on the above evaluation results, the multi-layer speech evaluation model outputs a local quality score for the corrected segment. This score, along with the text matching error and error time period, measures the quality of the corrected speech segment. The local quality score directly reflects the overall quality of the corrected segment and is a value between 0 and 1, where 1 represents perfect and 0 represents completely unacceptable. The text matching error measures the semantic consistency between the corrected segment and the original text, reflecting the semantic accuracy of the speech. The error time period identifies any remaining errors in the corrected segment; if it is empty, the correction is successful.

[0079] The local quality score corresponding to each corrected segment is normalized and weighted by confidence.

[0080] Normalization adjusts evaluation results from different dimensions to a uniform range (usually [0,1]) for comprehensive evaluation. Local quality scores and text matching errors are normalized to ensure their values ​​are within the [0,1] range. Normalization ensures the comparability of scores across dimensions, eliminating differences in units and magnitudes. Normalization methods can be linear normalization or other suitable normalization functions. Normalized scores can be directly used in subsequent weighted calculations.

[0081] Confidence weighting is a process of weighting the confidence scores for each dimension based on the evaluation model. Confidence reflects the reliability of the evaluation model's assessment results. By weighting the confidence scores for each dimension based on the evaluation model, the actual quality of the corrected segment can be more accurately reflected. Confidence can be determined through estimates of uncertainty within the model or by averaging multiple evaluations.

[0082] The local quality score, weighted by normalization and confidence, is used as the final segment audio score output. The segment audio score is the final score that comprehensively considers the local quality score, text matching error, and error time period. It is used to measure the overall quality of the corrected speech segment and comprehensively reflects the overall quality of the corrected speech segment in terms of semantic consistency, pronunciation accuracy, and speech naturalness.

[0083] This embodiment, through refined evaluation of the quality of specific corrected segments in the second speech, accurately measures the comprehensive performance of the corrected speech segments in terms of semantic consistency, pronunciation accuracy, and speech naturalness. Utilizing the multi-dimensional output of a multi-layer speech evaluation model, combined with normalization processing and confidence weighting, a final segment audio score is generated. This not only improves the optimization efficiency of the speech synthesis model but also ensures the quality and reliability of the generated speech, thus providing strong support for the continuous improvement of zero-shot speech synthesis models and ultimately enhancing the user experience in applications such as intelligent voice assistants and content generation platforms.

[0084] S106. Using the audio segment score as an instant reward signal, the model parameters of the zero-shot speech synthesis model are iteratively updated through the direct preference optimization algorithm until the speech generated by the zero-shot speech synthesis model meets the preset quality threshold.

[0085] The audio segment score comprehensively reflects the overall quality of the corrected speech segment in terms of semantic consistency, pronunciation accuracy, and speech naturalness. A high audio segment score indicates that the generated speech segment is of good quality, while a low audio segment score indicates that further optimization is needed.

[0086] The audio segment score is used as an immediate reward signal to guide the parameter update of the zero-shot speech synthesis model. The aim is to guide the zero-shot speech synthesis model to learn to generate higher quality speech that is more in line with human preferences. The ultimate goal is to enable the speech generated by the zero-shot speech synthesis model to stably meet the preset quality threshold.

[0087] The Direct Preference Optimization (DPO) algorithm is used to update the model parameters of a zero-shot speech synthesis model based on immediate reward signals. DPO is an optimization method based on human preference feedback, used to adjust the model parameters of a zero-shot speech synthesis model to make the generated speech more consistent with human preferences.

[0088] Further, preference pairs are constructed, including preferred samples and inferior samples; the audio segment score is used as an immediate reward signal, and a loss function based on the direct preference optimization algorithm is calculated, wherein the loss function is used to measure the difference between the probability of the zero-shot speech synthesis model generating the preferred sample and the probability of generating the inferior sample; by minimizing the loss function, the gradient of the model parameters is calculated, and the model parameters of the zero-shot speech synthesis model are updated using the gradient descent optimization algorithm, so that the zero-shot speech synthesis model is more inclined to generate the preferred sample.

[0089] In this embodiment, the DPO algorithm uses segment audio scores as an immediate reward signal to guide the updating of zero-shot speech synthesis model parameters. The Direct Preference Optimization algorithm adjusts the zero-shot speech synthesis model parameters by maximizing the reward signal, making the zero-shot speech synthesis model more inclined to generate high-quality speech.

[0090] Specifically, the audio segment score is used as the immediate reward signal in reinforcement learning. The higher the audio segment score, the better the quality of the locally corrected speech generated by the zero-shot speech synthesis model, and the zero-shot speech synthesis model should be positively incentivized. The optimization objective of the zero-shot speech synthesis model is to maximize the immediate reward signal, i.e., to maximize the audio segment score.

[0091] In the direct preference optimization algorithm, preference pairs need to be constructed. In this scheme, preference pairs consist of two parts: preferred samples and disliked samples. Preferred samples refer to the model behavior that generates high-quality speech segments; specifically, when the zero-shot speech synthesis model receives masked text, its internal parameter configuration can generate speech segments with high audio scores. Disliked samples refer to the model behavior that generates low-quality speech segments; specifically, the speech segments generated by the original parameter configuration of the zero-shot speech synthesis model (before optimization) have lower audio scores. Each local correction and evaluation process generates a set of data points (masked text, preferred samples, disliked samples, reward score), which are collected for the optimization and updating of the zero-shot speech synthesis model.

[0092] Direct preference optimization (DPO) is a method that optimizes a policy model by directly utilizing human (or evaluation) preference feedback without requiring an explicit reward model. Its core idea is to adjust model parameters by maximizing the log-likelihood probability of preferred samples relative to unfavorable samples, making the model more inclined to generate the preferred output.

[0093] The core of the DPO algorithm is a specific loss function that compares the probability of a zero-shot speech synthesis model generating a "preferred sample" with the probability of generating a "disadvantaged sample" given a masked text. Specifically, the loss function typically includes a log-sigmoid function, whose input is the difference between the log probability of a preferred sample and the log probability of a disadvantaged sample, multiplied by a coefficient weighted by the reward signal (audio segment rating). The loss value is small when the probability of a preferred sample is much greater than the probability of a disadvantaged sample; conversely, the loss value is large. By minimizing this loss function, the zero-shot speech synthesis model learns to adjust its parameters to increase the probability of generating high-scoring speech.

[0094] Calculate the gradient of the loss function with respect to the model parameters. This gradient reflects the sensitivity of the parameters to the reward signal, that is, the effect of small changes in the parameters on the reward signal, thus indicating how the model parameters should be adjusted to reduce the loss. Use a gradient descent optimizer (such as AdamW) to make small updates to the generated parameters of the model along the inverse direction of the gradient. The update formula is: (new parameters = old parameters - learning rate * gradient).

[0095] When updating parameters, it's necessary to consider hyperparameters such as the learning rate to ensure the direction and step size of parameter updates are reasonable. The learning rate is a crucial hyperparameter in optimization algorithms, determining the step size for each parameter update. An excessively high learning rate may cause the model parameters to update too quickly, preventing convergence; a learning rate that is too low may result in a slow optimization process. Furthermore, the DPO algorithm can be optimized by adjusting other hyperparameters, such as regularization terms and batch size.

[0096] This embodiment utilizes the Direct Preference Optimization (DPO) algorithm, employing segment audio scores as immediate reward signals to iteratively update the parameters of the zero-shot speech synthesis model. This significantly improves the quality of the generated speech, making it more aligned with human preferences. The DPO algorithm adjusts model parameters by constructing preference pairs (preferred and disliked samples), maximizing the generation probability of preferred samples while minimizing the loss function. This not only enhances the semantic consistency, pronunciation accuracy, and naturalness of the speech but also ensures the stability and efficiency of the optimization process through the appropriate setting of hyperparameters such as the learning rate.

[0097] In each iteration, the zero-shot speech synthesis model fine-tunes its parameters using a newly collected batch of preference data. As the iterations progress, the model gradually learns which speech features (such as clear pronunciation, natural intonation, and smooth rhythm) will receive higher audio segment scores, thus making it more likely to generate these features in subsequent segments.

[0098] In one embodiment, one or more quality thresholds can be preset as optimization targets for the zero-shot speech synthesis model. These quality thresholds may include segment audio scoring thresholds, error detection thresholds, and stability thresholds. For example, requirements may include ensuring that segment audio scores consistently exceed 0.9, that the evaluation model no longer detects error time periods, and that N consecutive generated speech samples meet the quality requirements.

[0099] Furthermore, after each iteration of updating the model parameters, a new speech sample is generated and its quality is evaluated to obtain a segment audio score of the speech sample. The segment audio score of the speech sample is compared with at least one preset quality threshold to determine whether the speech sample meets the quality requirements. If the segment audio score of the speech sample is less than the quality threshold, the next iteration is performed until the segment audio score of the speech sample is greater than or equal to the quality threshold.

[0100] Specifically, after each parameter update, a new speech sample is generated and evaluated to obtain a new audio segment score. This new audio segment score is compared to a preset quality threshold to determine if the generated speech meets the quality requirements. If the score reaches or exceeds the preset quality threshold, the zero-shot speech synthesis model is considered to have been optimized to a satisfactory level, and the optimization process can end.

[0101] If the audio score of the generated speech segment is lower than the preset quality threshold, the next iteration continues. During iteration, an upper limit on the number of iterations can be set to prevent the optimization process from running indefinitely. If the quality threshold is not met even after reaching the upper limit, it can be decided whether to stop optimization or adjust the optimization strategy based on the actual situation.

[0102] The optimization process ends when the audio score of the generated speech segment is greater than or equal to the preset quality threshold, or when the maximum number of iterations is reached. At this point, the model parameters of the zero-shot speech synthesis model have been iteratively updated using the direct preference optimization algorithm, and the quality of the generated speech has been improved, better meeting the preset quality requirements.

[0103] This embodiment ensures that the generated speech meets high-quality requirements across multiple dimensions by presetting multiple quality thresholds (such as segment audio scoring threshold, error detection threshold, and stability threshold). The optimization process also sets an upper limit on the number of iterations to prevent infinite iterations, thus improving the efficiency and reliability of the optimization.

[0104] This application provides a method for optimizing a speech synthesis model. The method generates first speech by inputting the original text into a zero-shot speech synthesis model, providing a foundational speech sample for subsequent quality assessment and optimization. The original text and the first speech are then input into a multi-layer speech evaluation model to assess the quality of the first speech, generating detailed evaluation results. These results not only provide quantitative indicators of speech quality but also accurately identify erroneous segments in the speech through multi-dimensional and fine-grained evaluation. Based on the speech evaluation results, masked text corresponding to the erroneous segments in the first speech is generated, achieving precise location and labeling of the erroneous parts, improving optimization accuracy, avoiding the high computational cost of traditional overall regeneration, and enhancing optimization efficiency. The masked text is then re-inputted into the zero-shot speech synthesis model to generate second speech, further improving the efficiency and quality of speech synthesis. The second speech is then evaluated using a multi-layer speech evaluation model to obtain segment audio scores, directly reflecting the optimized speech quality and providing clear feedback for adjusting model parameters, ensuring the correctness of the optimization direction. Using audio segment scores as immediate reward signals, the model parameters are iteratively updated through a direct preference optimization algorithm until the generated speech meets a preset quality threshold. By maximizing the reward signal, the zero-shot speech synthesis model is ensured to adjust its parameters in each iteration in the direction of improving speech quality, which significantly improves the efficiency and accuracy of zero-shot speech synthesis model optimization.

[0105] Please see Figure 4 , Figure 4 This is a schematic diagram of the current embodiment of a speech synthesis model optimization device provided in this application. The speech synthesis model optimization device is used to execute the aforementioned speech synthesis model optimization method.

[0106] like Figure 4 As shown, the optimization device 200 for the speech synthesis model includes: a first speech generation module 201, a first speech evaluation module 202, a masked text generation module 203, a second speech generation module 204, a second speech evaluation module 205, and a model parameter optimization module 206.

[0107] The first speech generation module 201 is used to input the original text into the zero-shot speech synthesis model to generate the first speech; The first speech evaluation module 202 is used to input the original text and the first speech into a multi-layer speech evaluation model, perform quality evaluation on the first speech through the multi-layer speech evaluation model, and generate a speech evaluation result. The mask text generation module 203 is used to generate mask text corresponding to the error period in the first speech based on the speech evaluation result; The second speech generation module 204 is used to input the masked text into the zero-sample speech synthesis model to generate the second speech; The second speech evaluation module 205 is used to evaluate the quality of the second speech through the multi-layer speech evaluation model and obtain the segment audio score corresponding to the second speech. The model parameter optimization module 206 is used to use the audio segment score as an instant reward signal and iteratively update the model parameters of the zero-shot speech synthesis model through a direct preference optimization algorithm until the speech generated by the zero-shot speech synthesis model meets a preset quality threshold.

[0108] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the device and each module described above can be referred to the corresponding processes in the aforementioned speech synthesis model optimization method embodiments, and will not be repeated here.

[0109] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0110] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0111] See Figure 5 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0112] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform an optimization method for any speech synthesis model.

[0113] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0114] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform optimization methods for any speech synthesis model.

[0115] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0117] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: The original text is input into the zero-shot speech synthesis model to generate the first speech; The original text and the first speech are input into a multi-layer speech evaluation model. The first speech is then evaluated for quality using the multi-layer speech evaluation model to generate a speech evaluation result. Based on the speech evaluation results, a mask text corresponding to the erroneous time period in the first speech is generated; The masked text is input into the zero-shot speech synthesis model to generate a second speech; The quality of the second speech is evaluated using the multi-layer speech evaluation model to obtain the audio score of the corresponding segment of the second speech. Using the audio segment score as an immediate reward signal, the model parameters of the zero-shot speech synthesis model are iteratively updated through a direct preference optimization algorithm until the speech generated by the zero-shot speech synthesis model meets a preset quality threshold.

[0118] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the speech synthesis model optimization methods provided in the embodiments of this application.

[0119] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the computer device.

[0120] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of optimizing a speech synthesis model, characterized by, The method includes: The original text is input into the zero-shot speech synthesis model to generate the first speech; The original text and the first speech are input into a multi-layer speech evaluation model. The first speech is then evaluated for quality using the multi-layer speech evaluation model to generate a speech evaluation result. Based on the speech evaluation results, a mask text corresponding to the erroneous time period in the first speech is generated; The masked text is input into the zero-shot speech synthesis model to generate a second speech; The quality of the second speech is evaluated using the multi-layer speech evaluation model to obtain the audio score of the corresponding segment of the second speech. Using the audio segment score as an immediate reward signal, the model parameters of the zero-shot speech synthesis model are iteratively updated through a direct preference optimization algorithm until the speech generated by the zero-shot speech synthesis model meets a preset quality threshold.

2. The method of optimizing a speech synthesis model of claim 1, wherein, The step of inputting the original text and the first speech into a multi-layer speech evaluation model, and performing a quality assessment of the first speech through the multi-layer speech evaluation model to generate a speech evaluation result includes: The original text and the first speech are input into a multi-layer speech evaluation model to generate speech semantic content features and phoneme sequences. The speech semantic content features and the phoneme sequence are input into the sequence evaluation network of the multi-layer speech evaluation model to perform phoneme speech semantic alignment and semantic extraction, and output the speech evaluation result.

3. The method of optimizing a speech synthesis model of claim 2, wherein, The multi-layer speech evaluation model includes a speech feature encoder and a text semantic encoder; The step of inputting the original text and the first speech into a multi-layer speech evaluation model to generate speech semantic content features and phoneme sequences includes: The original text is input into the text semantic encoder, and the first speech is input into the speech feature encoder; The original text is converted into the phoneme sequence using the text semantic encoder. The speech semantic content features are obtained by extracting features from the first speech using the speech feature encoder.

4. The method of optimizing a speech synthesis model of claim 1, wherein, The speech evaluation results include an overall audio quality score, text matching error, and error time periods; The step of generating mask text corresponding to the error periods in the first speech based on the speech evaluation result includes: Based on the error time period, determine the text segment in the original text that corresponds to the error time period; The text fragment is marked as a mask region to generate the masked text.

5. The method of optimizing a speech synthesis model of claim 1, wherein, The step of evaluating the quality of the second speech using the multi-layer speech evaluation model to obtain the segment audio score corresponding to the second speech includes: The second speech is input into the multi-layer speech evaluation model to obtain a local quality score only for the time period corresponding to the masked text; The local quality scores are normalized and weighted by confidence to output the audio score of the segment.

6. The method of optimizing a speech synthesis model of claim 1, wherein, The step of using the audio segment score as an immediate reward signal and iteratively updating the model parameters of the zero-shot speech synthesis model through a direct preference optimization algorithm includes: Construct preference pairs, which include preferred samples and unfavorable samples; The audio segment score is used as an instant reward signal, and a loss function based on the direct preference optimization algorithm is calculated, wherein the loss function is used to measure the difference between the probability of the zero-shot speech synthesis model generating the preferred sample and the probability of generating the undesired sample; By minimizing the loss function, the gradient of the model parameters is calculated, and the model parameters of the zero-shot speech synthesis model are updated using the gradient descent optimization algorithm, making the zero-shot speech synthesis model more inclined to generate the preferred samples.

7. The method of optimizing a speech synthesis model of claim 1, wherein, The iterative update of the model parameters of the zero-shot speech synthesis model until the speech generated by the zero-shot speech synthesis model meets a preset quality threshold includes: After each iteration of updating the model parameters, the speech samples are regenerated and their quality is evaluated to obtain the segment audio scores of the speech samples. The audio segment score of the speech sample is compared with at least one preset quality threshold to determine whether the speech sample meets the quality requirements. If the audio segment score of the speech sample is less than the quality threshold, the next iteration will continue until the audio segment score of the speech sample is greater than or equal to the quality threshold.

8. An apparatus for optimizing a speech synthesis model, characterized by The optimization device for the speech synthesis model includes: The first speech generation module is used to input the original text into the zero-shot speech synthesis model to generate the first speech; The first speech evaluation module is used to input the original text and the first speech into a multi-layer speech evaluation model, perform quality evaluation on the first speech through the multi-layer speech evaluation model, and generate a speech evaluation result. The mask text generation module is used to generate mask text corresponding to the error period in the first speech based on the speech evaluation result; The second speech generation module is used to input the masked text into the zero-sample speech synthesis model to generate the second speech; The second speech evaluation module is used to evaluate the quality of the second speech through the multi-layer speech evaluation model and obtain the audio score of the segment corresponding to the second speech. The model parameter optimization module is used to use the audio segment score as an instant reward signal and iteratively update the model parameters of the zero-shot speech synthesis model through a direct preference optimization algorithm until the speech generated by the zero-shot speech synthesis model meets a preset quality threshold.

9. A computer device, comprising: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the optimization method for the speech synthesis model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the optimization method for the speech synthesis model as described in any one of claims 1 to 7.

Images