Text-to-speech model adjustment method and device, speech synthesis method and device

Abstract

The embodiment of the present disclosure provides an adjustment method and device of a text-to-speech model, a speech synthesis method and device, and relates to the technical field of electric digital data processing. The adjustment method comprises the following steps: generating a first candidate audio sample set in a random sampling manner based on a text-to-speech (TTS) model; performing multi-dimensional evaluation on the first candidate audio sample set to filter out a first fine-tuning data set from the first candidate audio sample set; and fine-tuning the TTS model based on at least the first fine-tuning data set to obtain an adjusted TTS model, wherein the multi-dimensional evaluation comprises evaluating at least two of the following: a speech category, a timbre, a naturalness, a prosody and an audio quality of a sample in the first candidate audio sample set.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of electronic digital data processing, and more particularly to methods for adjusting text-to-speech models, apparatus for adjusting text-to-speech models, speech synthesis methods, speech synthesis apparatus, electronic devices, non-volatile computer-readable storage media, and computer program products. Background Technology

[0002] Text-to-Speech (TTS) technology refers to the technology of automatically converting text into natural speech.

[0003] In recent years, speech synthesis technology based on Large Language Model (LLM) (TTSLLM) has treated speech generation as a sequence-to-sequence generation task, capable of generating speech with specific timbre and style given text and voice prompts.

[0004] However, in some TTS LLM application scenarios, the number of audio samples used for training or fine-tuning the TTS model is small, which cannot meet the requirements for training or fine-tuning the TTS model. In addition, the quality of the audio samples is uneven, resulting in poor stability of speech synthesis based on the TTS model. Summary of the Invention

[0005] Embodiments of this disclosure provide speech synthesis methods, speech synthesis apparatuses, electronic devices, non-volatile computer-readable storage media, and computer program products that can at least partially solve the problems described above or other problems in the art.

[0006] According to a first aspect of this disclosure, a method for adjusting a text-to-speech (TTS) model is provided. The method includes: generating a first candidate audio sample set based on a text-to-speech (TTS) model using random sampling; performing a multi-dimensional evaluation on the first candidate audio sample set to select a first fine-tuning dataset from the first candidate audio sample set; and fine-tuning the TTS model based on at least the first fine-tuning dataset to obtain an adjusted TTS model, wherein the multi-dimensional evaluation includes evaluating at least two of the following: speech category, timbre, naturalness, prosody, and audio quality of the samples in the first candidate audio sample set.

[0007] According to a second aspect of this disclosure, a speech synthesis method is provided, the method comprising: inputting text to be synthesized into a TTS model to generate speech, wherein the TTS model is obtained by adjusting the text-to-speech model based on the first aspect and any possible implementation thereof.

[0008] According to a third aspect of this disclosure, an apparatus for adjusting a text-to-speech model is provided. The apparatus includes: a generation unit configured to generate a first candidate audio sample set based on a text-to-speech (TTS) model using random sampling; a filtering unit configured to perform multi-dimensional evaluation on the first candidate audio sample set to filter out a first fine-tuning dataset from the first candidate audio sample set; and a processing unit configured to fine-tune the TTS model based at least on the first fine-tuning dataset to obtain an adjusted TTS model, wherein the multi-dimensional evaluation includes evaluating at least two of the following: speech category, timbre, naturalness, prosody, and audio quality of the samples in the first candidate audio sample set.

[0009] According to a fourth aspect of this disclosure, a speech synthesis apparatus is provided, the apparatus comprising: a TTS model configured to generate speech based on input text to be synthesized, wherein the TTS model is obtained by adjusting a text-to-speech model based on the first aspect described above and any possible implementation thereof.

[0010] According to a fifth aspect of this disclosure, an electronic device is provided, comprising a processor and a memory, the processor being configured to implement the text-to-speech model adjustment method of the first aspect and any possible implementation thereof, as well as the speech synthesis method of the second aspect.

[0011] According to a sixth aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the text-to-speech model adjustment method of the first aspect and any possible implementation thereof, as well as the speech synthesis method of the second aspect.

[0012] According to the seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the text-to-speech model adjustment method of the first aspect and any possible implementation thereof, and the speech synthesis method of the second aspect.

[0013] The text-to-speech model adjustment method and apparatus, speech synthesis method and apparatus, electronic device, non-volatile computer-readable storage medium, and computer program product provided by the embodiments of this disclosure generate a first candidate audio sample set based on a TTS model using random sampling. Due to the diversity and randomness of the random sampling method, it ensures that a certain proportion of high-quality audio samples are present in the batch of audio samples generated by the TTS model (e.g., the first candidate audio sample set). By performing multi-dimensional evaluation on the samples in the first candidate audio sample set based on at least two of speech category, timbre, naturalness, prosody, and audio quality, high-quality audio samples (e.g., the first fine-tuning dataset) can be selected from the batch of audio samples. This achieves automated amplification and quality screening of fine-tuning data for the TTS model, and improves the efficiency of fine-tuning data acquisition and the objectivity of screening. When the number of audio samples used for training or fine-tuning is small, a sufficient number of high-quality audio samples can be obtained efficiently without increasing the training or fine-tuning cost of the TTS model, to meet the requirements for training or fine-tuning the TTS model.

[0014] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0015] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figures 1 to 4 These are flowcharts of a text-to-speech model adjustment method provided according to exemplary embodiments of this disclosure; Figure 5 This is a flowchart of a method for multi-dimensional evaluation of candidate audio samples provided according to an exemplary embodiment of this disclosure; Figures 6 to 7 These are flowcharts of a text-to-speech model adjustment method provided according to exemplary embodiments of this disclosure; Figure 8 This is a block diagram of a text-to-speech model adjustment apparatus provided according to an exemplary embodiment of the present disclosure; Figure 9 This is a schematic block diagram of an electronic device provided according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0017] The various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0018] The term “exemplary” as used herein means “serving as an example, implementation method, or illustration.” Any implementation method illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other implementation methods.

[0019] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, apparatuses, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0020] Some embodiments of this disclosure provide a method for adjusting a text-to-speech model. Figure 1 This is a flowchart of a text-to-speech model adjustment method 1000 provided according to an exemplary embodiment of the present disclosure.

[0021] like Figure 1 As shown, the text-to-speech model adjustment method 1000 includes: Step S1: Based on the TTS model, generate the first candidate audio sample set by random sampling.

[0022] Step S2: Perform a multi-dimensional evaluation on the first candidate audio sample set to select the first fine-tuning dataset from the first candidate audio sample set. The multi-dimensional evaluation includes evaluating at least two of the following: speech category, timbre, naturalness, prosody, and audio quality of the samples in the first candidate audio sample set.

[0023] Step S3: Fine-tune the TTS model based at least on the first fine-tuning dataset to obtain the adjusted TTS model.

[0024] A TTS model can be understood as a neural network model capable of converting input text into corresponding speech signals. Based on LLM (Limited Language Modeling), a TTS model can treat speech generation as a sequence-to-sequence modeling task, generating speech with specific timbre, style, or paralinguistic features given text and a Voice Prompt. Paralinguistic features can be understood as non-lexical semantic vocalizations. For example, the Voice Prompt may include reference audio.

[0025] Alternatively, the TTS model can be any autoregressive or diffusion speech synthesis model capable of achieving random sampling output by adjusting sampling parameters. In embodiments of this disclosure, the TTS model acts as a generator, generating candidate audio samples in batches through random sampling. This method of generating candidate audio samples in batches through random sampling can construct a first candidate audio sample set as well as other candidate audio sample sets used in subsequent embodiments of this disclosure.

[0026] During the training or fine-tuning of TTS models, audio sample sets in certain scenarios suffer from problems such as a small number of audio samples and inconsistent audio sample quality, making it difficult to meet the requirements for TTS model training or fine-tuning, thus leading to poor stability of speech synthesis based on the TTS model. For example, in scenarios of generating speech with a specific accent (e.g., Cantonese), the scarcity and high cost of high-quality recording data for that specific accent result in a limited number of available audio samples for the TTS model training or fine-tuning process in this scenario. In scenarios of generating speech with a specific style (e.g., broadcast voice, whisper), such speech requires professional actors to record in specific environments, making it difficult to obtain and resulting in a generally small scale of audio samples. In scenarios of generating paralinguistics, which can be understood as non-lexical semantic vocalizations (e.g., laughter, coughing, sighing, etc.), they account for a very small proportion in regular speech data, and the high cost of manual annotation leads to a severe shortage of high-quality paralinguistic samples.

[0027] In some training or fine-tuning schemes for related TTS models, the problems of small number of audio samples and inconsistent audio sample quality can be addressed by using two methods: "manual recording and annotation" and "large-scale data pre-training".

[0028] Specifically, taking the scenario of generating paralinguistics using a TTS model as an example, the method of generating paralinguistics through "manual recording and annotation" requires hiring professional voice actors to record various paralinguistic samples (e.g., laughter, crying, sighs, etc.) one by one in a recording studio environment, followed by manual listening tests for classification and cleaning. This method has low data collection efficiency. A professional voice actor can only effectively record a few hundred high-quality paralinguistic samples per day, and it is difficult to cover long-tail categories, such as awkward dry laughter or uncontrollable bursts of laughter. In addition, the efficiency of manual listening tests is also low; a quality inspector can only process about 1,000 audio samples per day, and the evaluation criteria are highly subjective, resulting in inconsistent data quality.

[0029] Taking the scenario of TTS model generating paralinguistics as an example, in the method of generating paralinguistics through "large-scale data pre-training," batches of audio data such as podcasts and audiobooks can be crawled from the Internet and labeled using automatic annotation tools, with the expectation that the TTS model will naturally learn paralinguistics generation from the large-scale audio data. However, in the above-mentioned batches of audio data, the proportion of paralinguistic segments is usually less than 1%, and the number of actually effective audio samples for TTS model training or fine-tuning is extremely limited. Moreover, the accuracy of automatic annotation is difficult to guarantee, resulting in inconsistent audio sample quality.

[0030] Therefore, in order to improve the quantity and quality of data used for training or fine-tuning TTS models, embodiments of this disclosure provide a text-to-speech model adjustment method 1000, which includes: generating a first candidate audio sample set based on a text-to-speech TTS model using random sampling; performing multi-dimensional evaluation on the first candidate audio sample set to select a first fine-tuning dataset from the first candidate audio sample set; and fine-tuning the TTS model based on at least the first fine-tuning dataset to obtain an adjusted TTS model, wherein the multi-dimensional evaluation includes evaluating at least two of the following: speech category, timbre, naturalness, prosody, and audio quality of the samples in the first candidate audio sample set.

[0031] The random sampling of TTS models is characterized by diversity and randomness. Specific methods to achieve random sampling include setting the temperature parameter of the TTS model to a value greater than 1 (e.g., the temperature parameter range is 1.0-1.2); setting the kernel sampling parameter to 0.9 to 1.0; and adding random noise during the sampling process of the TTS model. Random sampling of the TTS model can yield a batch of candidate audio samples, which can then be used to construct the first candidate audio sample set.

[0032] Furthermore, by setting a high temperature parameter for the TTS model, the model can exhibit greater randomness in generating candidate audio samples. While the overall quality of the candidate audio sample set is unstable, this ensures that at least a portion of the generated batch of candidate audio samples contains high-quality samples. For example, this portion of high-quality candidate audio samples constitutes approximately 30% of the total batch of candidate audio samples.

[0033] By evaluating at least two of the following dimensions—speech category, timbre, naturalness, prosody, and audio quality—from the first candidate audio sample set, a first fine-tuning dataset can be selected. Speech category can be understood as the categorical attribute of the speech signal expressed by the audio sample. For example, speech category may include paralinguistic categories, which may include non-lexical semantic vocalization types, such as laughter, crying, sighing, coughing, etc. Timbre can be understood as the spectral characteristics and perceptual attributes of sound, constituting the unique quality that distinguishes different speakers' voices. Naturalness, as a perceptual quality attribute, measures how closely synthesized speech resembles the naturalness of a human voice. Prosody can be understood as features such as rhythm, stress, and intonation patterns in the audio sample. For example, prosody can carry paralinguistic information such as emotion, attitude, and intention. Audio quality measures whether the audio sample has technical defects such as electronic sounds, discontinuity, distortion, and artifacts.

[0034] Therefore, through multi-dimensional comprehensive evaluation, high-quality samples that perform well in terms of speech category accuracy, timbre similarity, naturalness, prosodic naturalness, and audio technical quality can be automatically screened from the candidate audio sample set. This provides a reliable data foundation for the fine-tuning of the TTS model and effectively improves the stability and expressiveness of the model's generated paralinguistic audio.

[0035] The text-to-speech model adjustment method provided in this disclosure generates a first candidate audio sample set based on a TTS model using random sampling. Due to the diversity and randomness of the random sampling method, it ensures that a certain proportion of high-quality audio samples are present in the batch of audio samples generated by the TTS model (e.g., the first candidate audio sample set). By performing multi-dimensional evaluation on the first candidate audio sample set based on at least two of speech category, timbre, naturalness, prosody, and audio quality, the high-quality audio samples (e.g., the first fine-tuning dataset) can be selected from the batch of audio samples. This achieves automated amplification and quality screening of fine-tuning data for the TTS model, and improves the efficiency of data acquisition and the objectivity of screening. When the number of audio samples used for training or fine-tuning is small, a sufficient number of high-quality audio samples can be obtained efficiently without increasing the training or fine-tuning cost of the TTS model, meeting the requirements for training or fine-tuning the TTS model.

[0036] For example, in a scenario where a TTS model is used to generate a sub-language, the samples in the first candidate audio sample set and the first fine-tuning dataset may include sub-language audio samples, where the sub-language is a non-lexical semantic pronunciation.

[0037] Specifically, paralinguistic elements are non-linguistic acoustic elements in speech other than linguistic content (e.g., vocabulary, semantics), such as laughter, coughing, sighing, crying, and hiccups. These paralinguistic elements carry rich emotional and physiological information and play an important role in improving the naturalness and expressiveness of synthesized speech.

[0038] In scenarios where TTS models are used to generate sub-language, the first candidate audio sample set is a collection of a large number of sub-language candidate audio samples generated through random sampling. These candidate audio samples can cover multiple sub-language types, and the same type can have diverse variations, such as candidate audio samples of the same type with different intensities, durations, and contexts. The first fine-tuning dataset is a set of high-quality audio samples selected from the first candidate audio sample set through multi-dimensional evaluation of at least two of the following: speech category, timbre, naturalness, prosody, and audio quality. Therefore, it also includes multiple sub-language audio samples.

[0039] Furthermore, the technical solution disclosed herein is also applicable to other scenarios with problems such as a small number of samples and inconsistent quality, such as specific accent speech scenarios or specific style speech scenarios. In these scenarios, the samples in the first candidate audio sample set and the first fine-tuning dataset can be specific accent audio samples or specific style audio samples, respectively. The following description uses the scenario of TTS model generating sub-language as an example, but the scope of protection of this disclosure is not limited thereto.

[0040] Furthermore, taking the scenario of generating sub-language using a TTS model as an example, the other candidate audio sample sets and other fine-tuning datasets described below also include sub-language audio samples, and adopt the same generation and selection methods as the first candidate audio sample set and the first fine-tuning dataset. In other scenarios such as specific accent speech scenarios or specific style speech scenarios, the other candidate audio sample sets and other fine-tuning datasets described below can also include specific accent audio samples or specific style audio samples, and adopt the same generation and selection methods as the first candidate audio sample set and the first fine-tuning dataset.

[0041] The first candidate audio sample set and the first fine-tuning dataset include paralinguistic audio samples, which specifically addresses the technical problems of extremely low proportion of paralinguistic data in regular corpora, high cost of manual annotation, and severe lack of high-quality samples, providing sufficient and high-quality paralinguistic training data for the TTS model. The text-to-speech model adjustment scheme provided by the embodiments of this disclosure also includes other candidate audio sample sets and other fine-tuning datasets. These sample sets and datasets also include paralinguistic audio samples, ensuring the consistency and continuity of data types during the TTS model adjustment process. This allows the TTS model to consistently focus on the paralinguistic generation task during continuous optimization, avoiding training bias caused by mixed data types.

[0042] Figure 2 This is a flowchart of a text-to-speech model adjustment method 1000 provided according to an exemplary embodiment of the present disclosure.

[0043] like Figure 2 As shown, before step S1, which generates the first candidate audio sample set based on the TTS model using random sampling, the text-to-speech model adjustment method 1000 may further include: Step S0: Train the TTS model based on the seed dataset to train the TTS model's ability to generate corresponding audio based on the input speech category label. The seed dataset may include audio samples and speech category labels corresponding to the audio samples, which are used to indicate the target speech category.

[0044] It should be noted that in some embodiments of this disclosure, when the TTS model has the ability to generate first candidate audio samples that meet the requirements, the first candidate audio sample set can be directly generated according to the random sampling method. Therefore, step S0 can be omitted in this embodiment. For example, in the scenario of generating sub-language with a TTS model, based on the TTS model's ability to generate sub-language audio samples, step S1 can be directly executed to generate the first candidate audio sample set based on the TTS model using the random sampling method.

[0045] In some other embodiments of this disclosure, when the TTS model lacks the ability to generate first candidate audio samples that meet the requirements, a seed dataset can be obtained. The data type in the seed dataset can be consistent with the data type in the first candidate audio sample set. By training the TTS model based on the seed dataset in step S0, the TTS model can be equipped with or enhanced the ability to generate candidate audio samples, or in other words, the ability to generate corresponding audio based on the input speech category label. For example, in a scenario where a TTS model generates sub-language, the target speech category can be a sub-language category, such as laughter, crying, coughing, sneezing, etc. Since the TTS model lacks the ability to generate sub-language audio samples, the seed dataset can be used to train the TTS model, enabling or enhancing the TTS model to generate corresponding sub-language audio samples based on the input speech category label. The seed dataset may include sub-language audio samples (also called sub-language type audio samples) and corresponding speech category labels, which are used to indicate the corresponding sub-language category.

[0046] For example, the TTS model can be trained in a supervised manner based on a seed dataset. This supervised training may include: constructing training instructions by combining speech category labels and reference audio of the target speaker; inputting these training instructions into the TTS model to obtain predicted audio generated by the TTS model based on the current parameters; constructing a loss function based on the difference between the predicted audio and the corresponding audio samples in the seed dataset; and updating the parameters of the TTS model according to the value of the loss function until a preset training termination condition is met.

[0047] Training the TTS model based on the seed dataset allows the TTS model to have a generative foundation associated with the target speech category before random sampling. This guides the subsequent random sampling process to generate candidate samples in the direction of the speech category covered by the seed data, reducing invalid generation and increasing the proportion of valid samples in the candidate sample set.

[0048] For example, in a scenario where a TTS model is used to generate secondary language, the seed dataset may include audio samples and corresponding speech category labels, wherein the speech category labels may include at least one of the following: speech emotion expression labels and speech physiological response labels. Speech emotion expression labels may include at least one of the following: laughter labels, crying labels, sighing labels, and exclamation labels; and speech physiological response labels may include at least one of the following: coughing labels, sneezing labels, yawning labels, and hiccup labels.

[0049] In this scenario, the seed dataset can be constructed in the following ways: recorded by voice actors in a professional recording studio environment according to preset speech categories; or filtered from an existing speech database according to preset speech categories; or manually selected from internet audio resources that match preset speech categories. Optionally, the seed data can be recorded in triplet format, including the secondary language audio file; and the corresponding speech category label, such as a laughter label. <laugh>Cough label <cough>And so on; and speaker identifiers. For example, for a preset target speech category, the number of sub-language audio files belonging to different categories in the seed dataset can be 50 to 500 samples, and sub-language audio files of the same preset target speech category have speech category labels corresponding to that preset target speech category. Alternatively, the TTS model can be supervised or fine-tuned based on this seed dataset to enable the TTS model to acquire sub-language generation capabilities, thereby generating a first candidate audio sample set through random sampling in downstream steps.

[0050] In scenarios where TTS models are used to generate paralinguistics, by refining speech category labels into specific emotional expression and physiological response labels, TTS models can distinguish the acoustic features of different paralinguistic types, reduce category confusion during the generation process, and improve the semantic accuracy and natural expressiveness of the generated audio.

[0051] Figure 3 This is a flowchart of a text-to-speech model adjustment method 1000 provided according to an exemplary embodiment of the present disclosure.

[0052] like Figure 3 As shown, in some embodiments of this disclosure, step S1 may include: Step S1-1: Based on multiple speech category labels and the reference audio of the target speaker, construct generation instructions. The generation instructions are used to instruct the TTS model to generate audio samples that match the speech category labels and the timbre features of the target speaker in a random sampling manner. Input the generation instructions into the TTS model to obtain the first candidate audio sample set generated by the TTS model in a random sampling manner according to the generation instructions.

[0053] Specifically, the reference audio of the target speaker included in the generation instruction can be used to specify the timbre features that the synthesized speech should imitate; the reference audio may include audio of one or more target speakers, which can be selected as needed, and the scheme disclosed herein does not impose specific limitations.

[0054] The speech category labels included in the generation instructions can be used to specify the desired speech category generated from the synthesized speech (which can also be understood as audio samples) or the type of speech that the TTS model should attempt to generate. Alternatively, reference audio can be concatenated or combined with labels, and the generation instructions can be constructed based on the input format acceptable to the TTS model.

[0055] Optionally, the generation instructions may also include contextual text, which can provide a richer semantic environment for generating the first set of candidate audio samples. In other words, the contextual text can control the context in which the reference audio occurs, thereby making the candidate audio samples generated by the TTS model more natural.

[0056] For example, in a scenario where a TTS model is used to generate paralinguistics, multiple speech category labels may include at least one of speech emotion expression labels and speech physiological response labels. The speech emotion expression labels may include at least one of laughter, crying, sighing, and exclamation labels; and the speech physiological response labels may include at least one of coughing, sneezing, yawning, and hiccuping labels. For example, <laugh>Indicating laughter, <cough>This indicates a cough. Furthermore, the contextual text included in the generated instructions can be a natural language sentence that occurs simultaneously with the secondary language. It should be noted that the above secondary language scenario is merely an example, and the technical solution disclosed herein is not limited to secondary languages ​​but is applicable to any speech category label.

[0057] Furthermore, for each of the multiple speech category labels, a corresponding generation instruction can be constructed, and the TTS model can be independently invoked to perform random sampling to generate candidate audio samples corresponding to that speech category label, thereby obtaining the first candidate audio sample set based on the candidate audio samples of multiple speech category labels.

[0058] In this implementation, a generation instruction is constructed based on the speech category label and the reference audio of the target speaker, and the generation instruction is input into the TTS model. This allows the TTS model to be simultaneously constrained by two conditions during random sampling: "what speech category to generate" and "whose voice to imitate". This ensures that the candidate samples in the first candidate audio sample set generated by random sampling are controllable in both speech category and timbre dimensions, providing a candidate source that meets expectations for subsequent multi-dimensional evaluation and screening.

[0059] In addition, by using random sampling to generate the same generation instruction multiple times, and taking advantage of the high randomness of the TTS model's output characteristics, a large number of variant samples of the same speech category and the same timbre dimension can be generated in batches without human intervention. This effectively expands the candidate data scale of the first candidate audio sample set and makes up for the lack of training or fine-tuning data.

[0060] In some embodiments of this disclosure, step S2, which involves multi-dimensional evaluation of the first candidate audio sample set, may include performing at least two of the following: performing a semantic matching degree evaluation on the first candidate audio sample set to assess the consistency between the speech category expressed by the samples in the first candidate audio sample set and the speech category label, and generating a semantic matching quantification value; performing a timbre consistency evaluation on the first candidate audio sample set to assess the consistency between the timbre of the samples in the first candidate audio sample set and the timbre of the reference audio of the target speaker, and generating a timbre consistency quantification value; and performing an audio quality evaluation on the first candidate audio sample set to assess whether the samples in the first candidate audio sample set have technical defects, and generating an audio quality quantification value.

[0061] It should be noted that the above three evaluation dimensions are merely examples, and the multi-dimensional evaluation disclosed herein is not limited to these. In practical applications, evaluation dimensions can be added, reduced, or replaced according to the needs of specific scenarios, such as adding emotion intensity evaluation, speech rate evaluation, etc. In the embodiments of this disclosure, the multi-dimensional evaluation of any candidate audio sample set can adopt the same or corresponding dimension configuration, and the multi-dimensional evaluation under the same application scenario can adopt consistent evaluation dimensions to ensure the uniformity of the screening criteria.

[0062] Based on the diversity and randomness inherent in random sampling, a certain proportion of high-quality audio samples can be ensured in the batch of audio samples generated by the TTS model (e.g., the first candidate audio sample set). By performing the aforementioned multi-dimensional evaluation on this batch of audio samples, high-quality audio samples (e.g., the first fine-tuning dataset) can be selected, thereby achieving automated amplification and quality screening of the TTS model fine-tuning data.

[0063] For example, in a scenario where a TTS model is used to generate a secondary language, semantic matching evaluation can determine the speech category of the generated audio, such as whether it is "laughter" or "cough," to prevent speech category confusion; timbre consistency evaluation can ensure that the generated secondary language audio is consistent with the timbre of the target speaker, avoiding timbre drift; and audio quality evaluation can detect and remove low-quality samples with defects such as electronic sounds, discontinuity, and distortion.

[0064] Figure 4 This is a flowchart of a text-to-speech model adjustment method 1000 provided according to an exemplary embodiment of the present disclosure. Figure 5 This is a flowchart of a method for multi-dimensional evaluation of candidate audio samples provided according to an exemplary embodiment of this disclosure.

[0065] like Figure 4 and Figure 5 As shown, in some embodiments of this disclosure, step S2 may include: incorporating the sample into the first fine-tuning dataset based on the semantic matching quantization value of the sample being greater than or equal to a first threshold, the timbre consistency quantization value being greater than or equal to a second threshold, and the audio quality quantization value being greater than or equal to a third threshold.

[0066] Specifically, the first threshold, second threshold, and third threshold can be used to measure whether the audio samples in the first candidate audio sample set meet the minimum requirements for semantic matching, timbre consistency, and audio quality, respectively. During the evaluation process, cascading judgments can be made in the order of semantic matching degree, timbre consistency, and audio quality. If the quantization value of the current dimension does not reach the corresponding threshold, the audio sample can be directly judged as unqualified and the evaluation of subsequent dimensions can be terminated. Only when the quantization values ​​of all three dimensions reach or exceed the corresponding preset thresholds will the audio sample be regarded as a high-quality sample and selected into the first fine-tuning dataset.

[0067] It should be noted that the cascading judgment based on semantic matching degree, timbre consistency, and audio quality described above is for illustrative purposes only and does not constitute a limitation on the evaluation order. Regardless of the order used, the judgment logic is as follows: an audio sample is considered a high-quality sample and selected for inclusion in the first fine-tuning dataset only if the quantization values ​​of all three dimensions reach or exceed the corresponding preset thresholds; if the quantization value of any dimension of the audio sample does not reach the corresponding threshold, the audio sample can be discarded.

[0068] Furthermore, the three evaluation dimensions—semantic matching degree, timbre consistency, and audio quality—are merely examples, and the multi-dimensional evaluation disclosed herein is not limited to these. In practical applications, evaluation dimensions can be added, reduced, or replaced according to the specific needs of the scenario. Accordingly, for multi-dimensional evaluation using any dimension configuration, the selection criteria follow the following principle: in response to the audio sample's quantization value reaching or exceeding the corresponding threshold on all configured evaluation dimensions, the audio sample is included in the first fine-tuning dataset. Moreover, in the embodiments of this disclosure, for the first candidate audio sample set, and the second and third candidate audio sample sets described below, the multi-dimensional evaluation preferably uses the same or corresponding dimension configuration to ensure consistency of the selection criteria.

[0069] For example, the following will use steps S21 to S29 of multi-dimensional evaluation of the first candidate audio sample set as an example to describe the method of filtering candidate sets at different times to obtain the corresponding fine-tuning dataset.

[0070] In step S21, the first candidate audio sample set can be input into the trained screening model.

[0071] In step S22, each sample in the first candidate audio sample set can be evaluated in multiple dimensions, including semantic matching degree evaluation, timbre consistency evaluation and audio quality evaluation.

[0072] In step S23, for any sample in the first candidate audio sample set, it can be determined whether the semantic matching metric value of the sample is greater than or equal to the first threshold.

[0073] In step S24, in response to the semantic matching metric value of the sample being less than a first threshold, the sample is marked as an unqualified sample.

[0074] In step S25, in response to the semantic matching quantization value of the sample being greater than or equal to the first threshold, it is determined whether the timbre consistency quantization value of the sample is greater than or equal to the second threshold.

[0075] In step S26, in response to the timbre consistency quantization value of the sample being less than the second threshold, the sample is marked as an unqualified sample.

[0076] In step S27, in response to the timbre consistency quantization value of the sample being greater than or equal to the second threshold, it is determined whether the audio quality quantization value of the sample is greater than or equal to the third threshold.

[0077] In step S28, in response to the audio quality quantization value of the sample being less than the third threshold, the sample is marked as an unqualified sample.

[0078] In step S29, in response to the audio quality quantization value of the sample being greater than or equal to the third threshold, the sample is included in the first fine-tuning dataset.

[0079] By setting independent thresholds for multiple dimensions and performing joint judgments, it is ensured that samples included in the first fine-tuning dataset simultaneously meet preset standards in terms of semantics, timbre, and quality, avoiding the introduction of defective samples that might be introduced by screening based on a single dimension. Subsequent evaluations are terminated when any dimension fails to meet the threshold. This cascading judgment decision-making method reduces unnecessary computational overhead and improves screening efficiency. The entire process requires no manual intervention, achieving automated extraction of high-quality samples, ensuring the consistency and objectivity of the screening criteria, and providing a reliable data foundation for subsequent fine-tuning of the TTS model.

[0080] For example, step S2, which evaluates the first candidate audio sample set in multiple dimensions to select the first fine-tuned dataset from the first candidate audio sample set, may include: using a trained selection model to evaluate the first candidate audio sample set in multiple dimensions to select the first fine-tuned dataset from the first candidate audio sample set, wherein the selection model may include a multimodal model.

[0081] Specifically, a multimodal model can be understood as a neural network model capable of simultaneously processing and understanding multiple modal information, which may include audio, text, images, and other information. In the embodiments of this disclosure, the multimodal model has at least audio understanding and text understanding capabilities, and can receive audio samples and their corresponding speech category labels (or evaluation instructions) as input, and output the scoring results of the audio sample on each evaluation dimension.

[0082] In some embodiments of this disclosure, multi-dimensional evaluation of the first candidate audio sample set using a trained screening model may include: constructing a screening instruction based on the first candidate audio sample set and its corresponding speech category labels, wherein the screening instruction instructs the screening model to perform multi-dimensional evaluation on each sample in the first candidate audio sample set, wherein the multi-dimensional evaluation includes evaluating at least two of the sample’s speech category, timbre, naturalness, prosody, and audio quality, and generating corresponding quantization values ​​for each; inputting the screening instruction into the screening model, wherein the screening model can evaluate the samples in the first candidate audio sample set based on the screening instruction, so as to select a first fine-tuning dataset from the first candidate audio sample set.

[0083] It should be noted that the technical solutions disclosed herein do not limit the screening model to a unified multimodal model. In other embodiments, the screening model can also be composed of multiple specialized models: for example, audio classification models (e.g., PANNs (Pretrained Audio Neural Networks), AST (Audio Spectrogram Transformer)) can be used for semantic matching evaluation; speaker verification models (e.g., ECAPA-TDNN (Emphasized Channel Attention, Propagation and Aggregation-Time Delay Neural Network), WeSpeaker (A Research and Production oriented Speaker Embedding Learning Toolkit)) can be used for timbre consistency evaluation; and speech quality evaluation models (e.g., PESQ (Perceptual Evaluation of Speech Quality)) can be used for audio quality evaluation. The specific models mentioned above are merely examples, and those skilled in the art can choose other models with the same or similar functions to replace them according to actual needs. These alternative solutions all fall within the protection scope of this disclosure.

[0084] By using a trained screening model to perform multi-dimensional evaluation of the first candidate audio sample set, the automated multi-dimensional evaluation of candidate audio samples is achieved, eliminating the need for manual listening tests, significantly improving screening efficiency, and ensuring the consistency and objectivity of evaluation criteria. Furthermore, the same trained screening model can be used for the multi-dimensional evaluation of different candidate sets, such as the second and third candidate audio sample sets described below.

[0085] Furthermore, multimodal models possess cross-modal understanding capabilities, enabling them to process audio signals and text labels simultaneously. This allows for multi-dimensional evaluation within a single inference iteration, avoiding the system complexity and integration costs associated with connecting multiple dedicated models in series or parallel. Therefore, a unified multimodal model can be used for simplified deployment, or a combination of multiple dedicated models can be employed for refined evaluation across various dimensions, flexibly adapting to different application scenarios and resource constraints.

[0086] Figure 6 This is a flowchart of a text-to-speech model adjustment method 1000 provided according to an exemplary embodiment of the present disclosure.

[0087] like Figure 6 As shown, in some embodiments of this disclosure, step S3, which fine-tunes the TTS model based at least on the first fine-tuning dataset, may include: performing a first fine-tuning on the TTS model based on the seed dataset and the first fine-tuning dataset, wherein, in response to a first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being greater than or equal to a fourth threshold, an adjusted TTS model is obtained.

[0088] Specifically, in this embodiment, the text-to-speech model adjustment method 1000 may include steps S0 to S3, wherein step S3 may include: Step S3-1: Perform the first fine-tuning on the TTS model based on the seed dataset and the first fine-tuning dataset.

[0089] Step S3-2: Determine whether the first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set is greater than or equal to the fourth threshold.

[0090] Step S3-3: In response to the first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being greater than or equal to the fourth threshold, the adjusted TTS model is obtained.

[0091] By calculating the first ratio and determining whether it reaches the preset fourth threshold, a clear and quantifiable termination condition can be provided for the TTS model adjustment process. If the proportion of high-quality samples selected meets the target requirements, the TTS model that has completed the first fine-tuning can be considered to have sufficient generation capacity and does not need to be further fine-tuned, thus avoiding unnecessary fine-tuning computational overhead.

[0092] In this implementation, the TTS model is fine-tuned using both the seed dataset and the first fine-tuning dataset. This preserves the original high-quality anchor points from the seed data while incorporating newly added high-quality samples selected automatically. This allows the TTS model to balance basic and enhancement capabilities during fine-tuning, improving the stability and generation quality of the fine-tuned model. Furthermore, the entire fine-tuning and judgment process requires no manual intervention, achieving automated decision-making for TTS model adjustments and ensuring the consistency and objectivity of yield rate assessment.

[0093] Furthermore, in step S3-3, in response to the first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being greater than or equal to the fourth threshold, the adjusted TTS model (also known as the TTS model that has completed the first fine-tuning) is obtained, and the adjusted TTS model is output. Then, step S3-4 can be executed to output the first fine-tuning dataset and the seed dataset.

[0094] The seed dataset is sourced from professional recordings or rigorous screening, ensuring stable and reliable quality. The first fine-tuning dataset consists of high-quality samples selected through automated multi-dimensional evaluation. Combining both datasets provides a high-quality training or fine-tuning data resource for other TTS models, demonstrating significant reusability. When fine-tuning other TTS models, the seed dataset serves as a stable reference anchor, forming the training data together with the augmented samples from the first fine-tuning dataset. This helps maintain the stability of the generated quality during fine-tuning and prevents other TTS models from deviating from the expected speech category or timbre features when learning from the augmented data.

[0095] Furthermore, step S3-1, which performs the first fine-tuning of the TTS model based on the seed dataset and the first fine-tuning dataset, may include: performing the first fine-tuning of the TTS model based on the seed dataset and the first fine-tuning dataset. In some embodiments, this fine-tuning process may include: constructing a fine-tuning instruction by combining the speech category label and the reference audio of the target speaker; inputting the fine-tuning instruction into the TTS model to obtain the predicted audio generated by the TTS model according to the current parameters; constructing a loss function based on the difference between the predicted audio and the corresponding audio samples in the first fine-tuning dataset; updating the parameters of the TTS model according to the value of the loss function until a preset fine-tuning termination condition is met. The parameter update methods of other fine-tuning processes described below are similar, and will not be repeated here.

[0096] In addition, refer to again Figure 6 In some embodiments of this disclosure, the text-to-speech model adjustment method 1000 may further include: performing at least one round of iterative fine-tuning on the TTS model in response to a first ratio being less than a fourth threshold.

[0097] Specifically, in this embodiment, the text-to-speech model adjustment method 1000 may include steps S0 to S3, wherein step S3 may include: Step S3-1: Perform the first fine-tuning on the TTS model based on the seed dataset and the first fine-tuning dataset.

[0098] Step S3-2: Determine whether the first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set is greater than or equal to the fourth threshold.

[0099] Step S3-5: In response to the first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being less than the fourth threshold, perform at least one round of iterative fine-tuning on the TTS model.

[0100] By setting a fourth threshold as the target value for yield, at least one round of iterative fine-tuning is automatically triggered if the first ratio is not met. This allows the TTS model to continuously optimize until the generated quality meets the preset requirements, avoiding the performance issues caused by terminating the process after a single fine-tuning if the quality fails to meet the target. Furthermore, the triggering conditions for iterative fine-tuning are based on quantifiable objective indicators (e.g., a comparison between the first ratio and the fourth threshold), eliminating the need for manual judgment and achieving automated decision-making in the TTS model enhancement process.

[0101] Furthermore, in step S3-5, in response to the first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being less than the fourth threshold, after performing at least one round of iterative fine-tuning on the TTS model, step S3-6 can be executed to output the adjusted TTS model, and step S3-7 can be executed to output all the fine-tuning datasets and seed datasets used in the iteration process.

[0102] The seed dataset is sourced from professional recordings or rigorous screening, ensuring its stable and reliable quality. All fine-tuning datasets used in the iteration process are high-quality samples selected through automated multi-dimensional evaluation. The combined output of both can serve as high-quality training or fine-tuning data resources for training or fine-tuning other TTS models, demonstrating significant reusability. When subsequently fine-tuning other TTS models, the seed dataset can serve as a stable reference anchor, forming the training data together with the augmented samples from all the aforementioned fine-tuning datasets. This helps maintain the stability of the generated quality during fine-tuning and prevents other TTS models from deviating from the expected speech category or timbre features when learning from the augmented data.

[0103] Figure 7 This is a flowchart of a text-to-speech model adjustment method 1000 provided according to an exemplary embodiment of the present disclosure.

[0104] like Figure 7 As shown, in some embodiments of this disclosure, step S3-5, in response to a first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being less than a fourth threshold, may include performing at least one round of iterative fine-tuning on the TTS model: Step S3-8: Based on the TTS model that has completed the Nth fine-tuning, generate the N+1th candidate audio sample set by random sampling, where N is a positive integer greater than or equal to 1.

[0105] Steps S3-9 involve multi-dimensional evaluation of the N+1th candidate audio sample set to select the N+1th fine-tuning dataset from the N+1th candidate audio sample set.

[0106] Step S3-10: Based on the seed dataset, the first fine-tuning dataset, and the fine-tuning dataset generated in the iterative fine-tuning, perform the N+1th fine-tuning on the TTS model that has completed the Nth fine-tuning.

[0107] If the first ratio (or the yield rate of the first fine-tuning dataset) does not reach the fourth threshold, it indicates that the current TTS model's ability to generate high-quality samples does not yet meet the preset requirements. Based on this, the TTS model that has already undergone the first fine-tuning can be used as a new generation basis, and a second candidate audio sample set can be generated again using random sampling. This second candidate audio set is then subjected to a multi-dimensional evaluation with the same configuration described above to select the second fine-tuning dataset. The seed dataset, the first fine-tuning dataset, and the second fine-tuning dataset are then merged to perform a second fine-tuning on the TTS model that has already undergone the first fine-tuning.

[0108] Based on this, the N+1th candidate audio sample set is generated by random sampling based on the TTS model that has been fine-tuned for the Nth time. The N+1th candidate audio sample set is evaluated in multiple dimensions to select the N+1th fine-tuning dataset. Based on the seed dataset, the first fine-tuning dataset, and the cumulative results from the second fine-tuning dataset to the N+1th fine-tuning dataset, the N+1th fine-tuning is performed on the TTS model that has been fine-tuned for the Nth time.

[0109] By using the recursive method described above, the mechanism of the second round of iteration can be extended to any subsequent round, achieving adaptive continuation and termination of the iteration process. This approach is logically consistent and easy to implement. Furthermore, each round accumulates all historical high-quality samples for fine-tuning, ensuring that the TTS model does not forget previously learned generation patterns while continuously improving.

[0110] Based on the above technical solution, the implementation method of this disclosure has the following beneficial effects: by using the high-quality samples selected in the previous fine-tuning and the high-quality samples added in this iteration together for fine-tuning, the TTS model can learn more high-quality variants on the basis of existing improvements, thereby continuously improving the yield rate of generated samples. In addition, each iteration adopts the same mechanism of "generating candidate audio sets - selecting fine-tuning datasets - fine-tuning TTS models", without changing the processing flow or introducing additional supervision signals, realizing the progressive enhancement of the TTS model's capabilities, and having good operability and scalability.

[0111] Furthermore, to improve the above step S3-5 process of performing at least one round of iterative fine-tuning on the TTS model in response to the first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being less than a fourth threshold, in some embodiments of this disclosure, step S3-5 may further include: obtaining an adjusted TTS model in response to satisfying any of the following conditions, wherein the condition may include: the current ratio of the number of samples in the N+1th fine-tuning dataset to the number of samples in the N+1th candidate audio sample set being greater than or equal to the fourth threshold; the difference between the current ratio and the first ratio being less than a fifth threshold; and the number of fine-tuning operations performed so far being greater than or equal to the preset maximum number of fine-tuning operations.

[0112] Specifically, as an option, step S3-5 may include: Step S3-11: Determine whether the current ratio of the number of samples in the N+1th fine-tuning dataset to the number of samples in the N+1th candidate audio sample set is greater than or equal to the fourth threshold; determine whether the difference between the current ratio and the first ratio is less than the fifth threshold; determine whether the number of fine-tuning operations performed is greater than or equal to the preset maximum number of fine-tuning operations. In other words, step S3-11 determines whether one of the above three conditions is met.

[0113] If any of the above conditions are met, step S3-6 can be executed to output the adjusted TTS model, and step S3-7 can be executed to output all the fine-tuning datasets and seed datasets used in the iteration process.

[0114] During at least one round of iterative fine-tuning, after each round of fine-tuning, the yield rate corresponding to that round is calculated, or the ratio of the number of samples in the fine-tuning dataset to the number of samples in the candidate audio sample set. If this ratio reaches a preset fourth threshold, it indicates that the generation quality of the current TTS model meets the target requirements, and the iterative fine-tuning process can be terminated. Furthermore, to prevent the iterative fine-tuning process from failing to converge due to an excessively high fourth threshold, a convergence threshold condition is also set. If the yield rate improvement between two adjacent rounds, or the difference between the current round's yield rate and the previous round's yield rate, is less than a preset fifth threshold, it can be considered that the TTS model's capability improvement has reached saturation, and the benefits of continued iterative fine-tuning are limited, at which point the iteration is also terminated. In addition, the implementation method of this disclosure also sets a maximum number of fine-tuning attempts as a safety mechanism to prevent the iterative fine-tuning process from entering an infinite loop due to abnormal situations that cause the first two conditions to be unmet.

[0115] By setting an absolute threshold condition (fourth threshold) during at least one round of iterative fine-tuning, a clear quality objective is provided for iteration termination, ensuring that the final output TTS model has sufficient high-quality sample generation capability. Setting a convergence threshold condition (fifth threshold) avoids a large number of invalid iterations due to slow quality improvement, saving computational resources and ensuring that the TTS model terminates promptly when performance saturates. Setting a preset maximum number of fine-tuning iterations as a safety mechanism prevents infinite loops caused by unreasonable threshold settings or data anomalies, ensuring the robustness and controllability of the entire adjustment process. Therefore, the strategy of "terminating upon meeting any condition" balances quality objectives, efficiency assurance, and safety, achieving automation of the iterative fine-tuning process and adaptive termination decisions without manual intervention.

[0116] For example, in some embodiments of this disclosure, the iterative fine-tuning process may continue even if the above three conditions are not met.

[0117] For example, in response to the failure to meet any condition, a third candidate audio sample set is generated by random sampling based on the TTS model that has completed the second fine-tuning; the third candidate audio sample set is evaluated in multiple dimensions to select the third fine-tuning dataset from the third candidate audio sample set; and a third fine-tuning is performed on the TTS model that has completed the second fine-tuning based on the seed dataset, the first fine-tuning dataset, the second fine-tuning dataset, and the third fine-tuning dataset.

[0118] Specifically, if none of the termination conditions in the current iteration are triggered, it indicates that the TTS model performance has not yet reached the target and there is still room for improvement. At this point, the currently fine-tuned TTS model is used as the basis for generation, and the mechanism of "generating candidate audio sets - filtering fine-tuning datasets - fine-tuning TTS models" is executed again. The high-quality samples added in this round (e.g., the third fine-tuning dataset) are merged with the fine-tuning datasets accumulated in previous rounds (seed dataset, first and second fine-tuning datasets) for the next round of fine-tuning.

[0119] In this way, by automatically entering the next iteration when the termination condition is not met, the TTS model can continuously optimize itself using newly added high-quality samples until its performance reaches the target or converges, thus achieving a gradual improvement in capabilities. Each iteration incorporates all historically accumulated high-quality samples into the training data, ensuring that the TTS model does not forget the high-quality generation patterns it has mastered during the learning process and continuously absorbs new high-quality variants, which helps to improve the stability and generalization ability of the TTS model.

[0120] In some embodiments of this disclosure, step S3, which is based at least on the first fine-tuning dataset, may further include fine-tuning the TTS model based on the first fine-tuning dataset and the general speech dataset, wherein the general speech dataset is a training sample set collected from a multi-category speech data source, and the multi-category speech data source includes at least one of the following: a Mandarin multi-speaker speech data source, a multi-dialect general speech data source, and a multi-language basic speech data source.

[0121] Specifically, a general speech dataset can include basic speech samples from different speakers, accents, and languages. These samples typically do not include specific target speech categories, such as paralinguistics, specific accents, or specific styles, but can help TTS models maintain or enhance their general speech synthesis capabilities.

[0122] During fine-tuning, the first fine-tuning dataset can be mixed with a general speech dataset. This allows the TTS model to learn specific category generation capabilities while avoiding forgetting or degrading its original basic speech synthesis capabilities.

[0123] It should be noted that the fine-tuning method based on the general speech dataset described above is not limited to the first fine-tuning. In the embodiments of this disclosure, this method can be used in any fine-tuning process during at least one round of iterative fine-tuning, or in other words, the TTS model can be fine-tuned based on a combination of the currently accumulated fine-tuning dataset (e.g., seed dataset, first fine-tuning dataset, second fine-tuning dataset, etc.) and the general speech dataset. Whether to use this method and the proportion of use can be determined according to the actual needs of the scenario.

[0124] refer to Figures 1 to 7 Some embodiments of this disclosure provide a speech synthesis method, which includes: inputting text to be synthesized into a TTS model to generate speech, wherein the TTS model can be obtained by adjusting the text-to-speech model adjustment method 1000 described in any of the above embodiments.

[0125] As described above, in the text-to-speech model adjustment method 1000, a first candidate audio sample set is generated based on the initial TTS model using random sampling. Due to the diversity and randomness of the random sampling method, it ensures that a certain proportion of high-quality audio samples are present in the batch of audio samples generated by the initial TTS model (e.g., the first candidate audio sample set). By performing multi-dimensional evaluation on the first candidate audio sample set based on at least two of speech category, timbre, naturalness, prosody, and audio quality, high-quality audio samples (e.g., the first fine-tuning dataset) can be selected from the batch of audio samples. This achieves automated amplification and quality screening of fine-tuning data for the initial TTS model, and improves the efficiency of data acquisition and the objectivity of screening. When the number of audio samples used for training or fine-tuning is small, a sufficient number of high-quality audio samples can be efficiently obtained without increasing the training or fine-tuning cost of the initial TTS model, meeting the requirements for training or fine-tuning the initial TTS model.

[0126] After adjusting the initial TTS model using the text-to-speech model adjustment method 1000 described in any of the above embodiments, the obtained TTS model can be applied in speech synthesis scenarios where training or fine-tuning data is scarce. For example, in scenarios such as generating paralinguistics, specific accent speech, or specific style speech, the text to be synthesized can be input into the adjusted TTS model to generate synthesized speech with high semantic matching degree, high timbre consistency, and high audio quality.

[0127] Since this TTS model is obtained through the text-to-speech model adjustment method 1000, the aforementioned speech synthesis method naturally possesses all the advantages brought by the text-to-speech model adjustment method 1000. For example, it can achieve stable generation of high-quality speech even under conditions of scarce training or fine-tuning data; it can avoid problems such as category confusion, timbre drift, and quality degradation; and the entire synthesis process does not require additional filtering or correction steps, maintaining a simple and efficient inference flow. Other beneficial effects of the speech synthesis method will not be elaborated upon here.

[0128] Figure 8 This is a block diagram of a text-to-speech model adjustment device 2000 provided according to an exemplary embodiment of the present disclosure.

[0129] like Figure 8 As shown, some embodiments of this disclosure provide a text-to-speech model adjustment device 2000, which includes: a generation unit 100, a filtering unit 200, and a processing unit 300. The generation unit 100 is configured to generate a first candidate audio sample set based on a text-to-speech (TTS) model using random sampling. The filtering unit 200 is configured to perform multi-dimensional evaluation on the first candidate audio sample set to select a first fine-tuning dataset from the first candidate audio sample set. The multi-dimensional evaluation includes evaluating at least two of the following: speech category, timbre, naturalness, prosody, and audio quality of the samples in the first candidate audio sample set. The processing unit 300 is configured to fine-tune the TTS model based at least on the first fine-tuning dataset to obtain an adjusted TTS model.

[0130] The random sampling of TTS models is characterized by diversity and randomness. Specific methods to achieve random sampling include setting the temperature parameter of the TTS model to a value greater than 1 (e.g., the temperature parameter range is 1.0-1.2); setting the kernel sampling parameter to 0.9 to 1.0; and adding random noise during the sampling process of the TTS model. Random sampling of the TTS model can yield a batch of candidate audio samples, which can then be used to construct the first candidate audio sample set.

[0131] Furthermore, by setting a high temperature parameter for the TTS model, the model can exhibit greater randomness in generating candidate audio samples. While the overall quality of the candidate audio sample set is unstable, this ensures that at least a portion of the generated batch of candidate audio samples contains high-quality samples. For example, this portion of high-quality candidate audio samples constitutes approximately 30% of the total batch of candidate audio samples.

[0132] By evaluating at least two of the following dimensions—speech category, timbre, naturalness, prosody, and audio quality—from the first candidate audio sample set, a first fine-tuning dataset can be selected. Speech category can be understood as the categorical attribute of the speech signal expressed by the audio sample. For example, speech category may include paralinguistic categories, which may include non-lexical semantic vocalization types, such as laughter, crying, sighing, coughing, etc. Timbre can be understood as the spectral characteristics and perceptual attributes of sound, constituting the unique quality that distinguishes different speakers' voices. Naturalness, as a perceptual quality attribute, measures how closely synthesized speech resembles the naturalness of a human voice. Prosody can be understood as features such as rhythm, stress, and intonation patterns in the audio sample. For example, prosody can carry paralinguistic information such as emotion, attitude, and intention. Audio quality measures whether the audio sample has technical defects such as electronic sounds, discontinuity, distortion, and artifacts.

[0133] Therefore, through multi-dimensional comprehensive evaluation, high-quality samples that perform well in terms of speech category accuracy, timbre similarity, naturalness, prosodic naturalness, and audio technical quality can be automatically screened from the candidate audio sample set. This provides a reliable data foundation for the fine-tuning of the TTS model and effectively improves the stability and expressiveness of the model's generated paralinguistic audio.

[0134] The text-to-speech model adjustment scheme provided by this disclosure generates a first candidate audio sample set based on a TTS model using random sampling. Due to the diversity and randomness of the random sampling method, it ensures that a certain proportion of high-quality audio samples are present in the batch of audio samples generated by the TTS model (e.g., the first candidate audio sample set). By performing multi-dimensional evaluation on the first candidate audio sample set based on at least two of speech category, timbre, naturalness, prosody, and audio quality, high-quality audio samples (e.g., the first fine-tuning dataset) can be selected from the batch of audio samples. This achieves automated amplification and quality screening of fine-tuning data for the TTS model, and improves the efficiency of data acquisition and the objectivity of screening. Even when the number of audio samples used for training or fine-tuning is small, a sufficient number of high-quality audio samples can be efficiently obtained without increasing the training or fine-tuning cost of the TTS model, meeting the requirements for training or fine-tuning the TTS model.

[0135] In some embodiments of this disclosure, the processing unit 300 may be configured to: train a TTS model based on a seed dataset before generating a first candidate audio sample set according to a generation instruction in a random sampling manner, so as to train the TTS model to generate corresponding audio based on the input speech category label, wherein the seed dataset may include audio samples and speech category labels corresponding to the audio samples, the speech category labels being used to indicate the target speech category.

[0136] Training the TTS model based on the seed dataset allows the TTS model to have a generative foundation associated with the target speech category before random sampling. This guides the subsequent random sampling process to generate candidate samples in the direction of the speech category covered by the seed data, reducing invalid generation and increasing the proportion of valid samples in the candidate sample set.

[0137] It should be noted that in some embodiments of this disclosure, when the TTS model has the ability to generate first candidate audio samples that meet the requirements, the first candidate audio sample set can be generated directly according to the random sampling method.

[0138] In some embodiments of this disclosure, the voice category label may include at least one of the following: voice emotion expression label and voice physiological response label, wherein the voice emotion expression label includes at least one of the following: laughter label, crying label, sighing label and exclamation label; and the voice physiological response label includes at least one of the following: cough label, sneezing label, yawning label and hiccup label.

[0139] In scenarios where a TTS model is used to generate a secondary language, the seed dataset can be constructed in the following ways: recorded by voice actors in a professional recording studio environment according to preset speech categories; or filtered from an existing speech database according to preset speech categories; or manually selected from internet audio resources that match preset speech categories. Optionally, the seed data can be recorded in a triplet format, including the secondary language audio file; and the corresponding speech category label, such as a laughter label. <laugh>Cough label <cough>And so on; and speaker identifiers. For example, for a preset target speech category, the number of sub-language audio files belonging to different categories in the seed dataset can be 50 to 500 samples, and sub-language audio files of the same preset target speech category have speech category labels corresponding to that preset speech category. Alternatively, the TTS model can be supervised or fine-tuned based on this seed dataset to enable the TTS model to acquire sub-language generation capabilities, thereby generating a first candidate audio sample set through random sampling in downstream steps.

[0140] In scenarios where TTS models are used to generate paralinguistics, by refining speech category labels into specific emotional expression and physiological response labels, TTS models can distinguish the acoustic features of different paralinguistic types, reduce category confusion during the generation process, and improve the semantic accuracy and natural expressiveness of the generated audio.

[0141] In some embodiments of this disclosure, the generation unit 100 may also be configured to: construct a generation instruction based on multiple speech category labels and reference audio of the target speaker, the generation instruction being used to instruct the TTS model to generate audio samples that match the speech category labels and the timbre features of the target speaker in a random sampling manner; and input the generation instruction into the TTS model to obtain a first candidate audio sample set generated by the TTS model in a random sampling manner according to the generation instruction.

[0142] The reference audio of the target speaker included in the generation instruction can be used to specify the timbre features that the synthesized speech should imitate; the reference audio may include audio of one or more target speakers, which can be selected as needed, and the scheme disclosed herein does not impose specific limitations.

[0143] The speech category labels included in the generation instructions can be used to specify the desired speech category for the synthesized speech or the type of speech the TTS model should attempt to generate. Alternatively, reference audio can be concatenated or combined with labels, and the generation instructions can be constructed based on input formats acceptable to the TTS model.

[0144] Optionally, the generation instructions may also include contextual text, which can provide a richer semantic environment for generating the first candidate audio sample set. In other words, the contextual text can be used to control the occurrence context of the reference audio, thereby making the candidate audio samples generated by the TTS model more natural. The contextual text included in the generation instructions can be a natural language sentence that occurs simultaneously with the paralinguistics. It should be noted that the above paralinguistic scenario is merely an example, and the technical solution disclosed herein is not limited to paralinguistics but is applicable to any speech category label.

[0145] Furthermore, for each of the multiple speech category labels, a corresponding generation instruction can be constructed, and the TTS model can be independently invoked to perform random sampling to generate candidate audio samples corresponding to that speech category label, thereby obtaining the first candidate audio sample set based on the candidate audio samples of multiple speech category labels.

[0146] In this implementation, a generation instruction is constructed based on the speech category label and the reference audio of the target speaker, and the generation instruction is input into the TTS model. This allows the TTS model to be simultaneously constrained by two conditions during random sampling: "what speech category to generate" and "whose voice to imitate". This ensures that the candidate samples in the first candidate audio sample set generated by random sampling are controllable in both speech category and timbre dimensions, providing a candidate source that meets expectations for subsequent multi-dimensional evaluation and screening.

[0147] In addition, by using random sampling to generate the same generation instruction multiple times, and taking advantage of the high randomness of the TTS model's output characteristics, a large number of variant samples of the same speech category and the same timbre dimension can be generated in batches without human intervention. This effectively expands the candidate data scale of the first candidate audio sample set and makes up for the lack of training or fine-tuning data.

[0148] In some embodiments of this disclosure, step S2, which involves multi-dimensional evaluation of the first candidate audio sample set, may include performing at least two of the following: performing a semantic matching degree evaluation on the first candidate audio sample set to assess the consistency between the speech category expressed by the samples in the first candidate audio sample set and the speech category label, and generating a semantic matching quantification value; performing a timbre consistency evaluation on the first candidate audio sample set to assess the consistency between the timbre of the samples in the first candidate audio sample set and the timbre of the reference audio of the target speaker, and generating a timbre consistency quantification value; and performing an audio quality evaluation on the first candidate audio sample set to assess whether the samples in the first candidate audio sample set have technical defects, and generating an audio quality quantification value.

[0149] It should be noted that the above three evaluation dimensions are merely examples, and the multi-dimensional evaluation disclosed herein is not limited to these. In practical applications, evaluation dimensions can be added, reduced, or replaced according to the needs of specific scenarios, such as adding emotion intensity evaluation, speech rate evaluation, etc. In the embodiments of this disclosure, the multi-dimensional evaluation of any candidate audio sample set can adopt the same or corresponding dimension configuration, and the multi-dimensional evaluation under the same application scenario can adopt consistent evaluation dimensions to ensure the uniformity of the screening criteria.

[0150] Based on the diversity and randomness inherent in random sampling, a certain proportion of high-quality audio samples can be ensured in the batch of audio samples generated by the TTS model (e.g., the first candidate audio sample set). By performing the aforementioned multi-dimensional evaluation on this batch of audio samples, high-quality audio samples (e.g., the first fine-tuning dataset) can be selected, thereby achieving automated amplification and quality screening of the TTS model fine-tuning data.

[0151] For example, in a scenario where a TTS model is used to generate a secondary language, semantic matching evaluation can determine the speech category of the generated audio, such as whether it is "laughter" or "cough," to prevent speech category confusion; timbre consistency evaluation can ensure that the generated secondary language audio is consistent with the timbre of the target speaker, avoiding timbre drift; and audio quality evaluation can detect and remove low-quality samples with defects such as electronic sounds, discontinuity, and distortion.

[0152] In some embodiments of this disclosure, the filtering unit 200 may also be configured to include the sample in the first fine-tuning dataset based on the sample's semantic matching quantization value being greater than or equal to a first threshold, the timbre consistency quantization value being greater than or equal to a second threshold, and the audio quality quantization value being greater than or equal to a third threshold.

[0153] The first, second, and third thresholds are used to measure whether audio samples in the first candidate audio sample set meet the minimum requirements for semantic matching, timbre consistency, and audio quality, respectively. During the evaluation process, cascading judgments can be made in the order of semantic matching degree, timbre consistency, and audio quality. If the quantization value of the current dimension does not reach the corresponding threshold, the audio sample can be directly judged as unqualified and the evaluation of subsequent dimensions can be terminated. Only when the quantization values ​​of all three dimensions reach or exceed the corresponding preset thresholds will the audio sample be considered a high-quality sample and selected into the first fine-tuning dataset.

[0154] It should be noted that the cascading judgment based on semantic matching degree, timbre consistency, and audio quality described above is for illustrative purposes only and does not constitute a limitation on the evaluation order. Regardless of the order used, the judgment logic is as follows: an audio sample is considered a high-quality sample and selected for inclusion in the first fine-tuning dataset only if the quantization values ​​of all three dimensions reach or exceed the corresponding preset thresholds; if the quantization value of any dimension of the audio sample does not reach the corresponding threshold, the audio sample can be discarded.

[0155] By setting independent thresholds for multiple dimensions and performing joint judgments, it is ensured that samples included in the first fine-tuning dataset simultaneously meet preset standards in terms of semantics, timbre, and quality, avoiding the introduction of defective samples that might be introduced by screening based on a single dimension. Subsequent evaluations are terminated when any dimension fails to meet the threshold. This cascading judgment decision-making method reduces unnecessary computational overhead and improves screening efficiency. The entire process requires no manual intervention, achieving automated extraction of high-quality samples, ensuring the consistency and objectivity of the screening criteria, and providing a reliable data foundation for subsequent fine-tuning of the TTS model.

[0156] In some embodiments of this disclosure, the screening unit 200 may also be configured to: perform multi-dimensional evaluation on the first candidate audio sample set using a trained screening model to select a first fine-tuned dataset from the first candidate audio sample set, wherein the screening model may include a multimodal model.

[0157] A multimodal model can be understood as a neural network model capable of simultaneously processing and understanding multiple modalities of information, including audio, text, and images. In embodiments of this disclosure, the multimodal model possesses at least audio and text understanding capabilities, and can receive audio samples and their corresponding speech category labels (or evaluation instructions) as input, and output scores for the audio sample across various evaluation dimensions. However, the technical solution of this disclosure does not limit the selection model to a unified multimodal model. In other embodiments, the selection model can also be composed of a combination of multiple specialized models.

[0158] In some embodiments of this disclosure, the screening unit 200 may also be configured to: construct a screening instruction based on a first candidate audio sample set and its corresponding speech category labels, the screening instruction being used to instruct the screening model to perform a multi-dimensional evaluation on each sample in the first candidate audio sample set, the multi-dimensional evaluation including evaluating at least two of the speech category, timbre, naturalness, prosody and audio quality of the sample, and generating corresponding quantization values ​​respectively; input the screening instruction into the screening model, the screening model being able to evaluate the samples in the first candidate audio sample set based on the screening instruction, so as to screen out a first fine-tuning dataset from the first candidate audio sample set.

[0159] Multimodal models possess cross-modal understanding capabilities, enabling them to process audio signals and text labels simultaneously. They complete multi-dimensional evaluations in a single inference, avoiding the system complexity and integration costs associated with connecting multiple dedicated models in series or parallel. Therefore, a unified multimodal model can be used for simple deployment, or a combination of multiple dedicated models can be chosen for refined evaluation of each dimension, flexibly adapting to different application scenarios and resource constraints.

[0160] In some embodiments of this disclosure, the samples in the first candidate audio sample set and the first fine-tuning dataset may include paralinguistic audio samples, where paralinguistics is non-lexical semantic vocalization.

[0161] Paralinguistic elements are non-linguistic acoustic elements in speech other than linguistic content (e.g., vocabulary, semantics), such as laughter, coughing, sighing, crying, and hiccups. These paralinguistic elements carry rich emotional and physiological information and play an important role in improving the naturalness and expressiveness of synthesized speech.

[0162] In scenarios where TTS models are used to generate sub-languages, the first candidate audio sample set is a collection of a large number of sub-language candidate audio samples generated through random sampling. These candidate audio samples can cover multiple sub-language types, and the same type can have diverse variations, such as candidate audio samples of the same type with different intensities, durations, and contexts. The first fine-tuning dataset is a collection of high-quality audio samples selected from the first candidate audio sample set through multi-dimensional evaluation, and therefore it also includes multiple sub-language audio samples.

[0163] Furthermore, the technical solution disclosed herein is also applicable to other scenarios with problems such as a small number of samples and inconsistent quality, such as specific accent speech scenarios or specific style speech scenarios. In these scenarios, the samples in the first candidate audio sample set and the first fine-tuning dataset can be specific accent audio samples or specific style audio samples, respectively.

[0164] In the scenario of generating sub-language using a TTS model as an example, the other candidate audio sample sets and other fine-tuning datasets involved in the embodiments of this disclosure can also include sub-language audio samples, and adopt the same generation and selection methods as the first candidate audio sample set and the first fine-tuning dataset. In other scenarios such as specific accent speech scenarios or specific style speech scenarios, these other candidate audio sample sets and other fine-tuning datasets can also include specific accent audio samples or specific style audio samples, and adopt the same generation and selection methods as the first candidate audio sample set and the first fine-tuning dataset.

[0165] The first candidate audio sample set and the first fine-tuning dataset include paralinguistic audio samples, which specifically addresses the technical problems of extremely low proportion of paralinguistic data in regular corpora, high cost of manual annotation, and severe lack of high-quality samples, providing sufficient and high-quality paralinguistic training data for the TTS model. The text-to-speech model adjustment scheme provided by the embodiments of this disclosure also includes other candidate audio sample sets and other fine-tuning datasets, both of which also include paralinguistic audio samples. This ensures the consistency and continuity of data types during the TTS model adjustment process, allowing the TTS model to always focus on the paralinguistic generation task during continuous optimization and avoiding training bias caused by mixed data types.

[0166] In some embodiments of this disclosure, the processing unit 300 may also be configured to: perform a first fine-tuning of the TTS model based on the seed dataset and the first fine-tuning dataset, wherein the adjusted TTS model is obtained in response to a first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being greater than or equal to a fourth threshold.

[0167] By calculating the first ratio and determining whether it reaches the preset fourth threshold, a clear and quantifiable termination condition can be provided for the TTS model adjustment process. If the proportion of high-quality samples selected meets the target requirements, the TTS model that has completed the first fine-tuning can be considered to have sufficient generation capacity and does not need to be further fine-tuned, thus avoiding unnecessary fine-tuning computational overhead.

[0168] In this implementation, the TTS model is fine-tuned using both the seed dataset and the first fine-tuning dataset. This preserves the original high-quality anchor points from the seed data while incorporating newly added high-quality samples selected automatically. This allows the TTS model to balance basic and enhancement capabilities during fine-tuning, improving the stability and generation quality of the fine-tuned model. Furthermore, the entire fine-tuning and judgment process requires no manual intervention, achieving automated decision-making for TTS model adjustments and ensuring the consistency and objectivity of yield rate assessment.

[0169] In some embodiments of this disclosure, the processing unit 300 may also be configured to perform at least one round of iterative fine-tuning on the TTS model in response to a first ratio of the number of samples in the first fine-tuning dataset to the number of samples in the first candidate audio sample set being less than a fourth threshold.

[0170] By setting a fourth threshold as the target value for yield, at least one round of iterative fine-tuning is automatically triggered if the first ratio is not met. This allows the TTS model to continuously optimize until the generated quality meets the preset requirements, avoiding the performance issues caused by terminating the process after a single fine-tuning if the quality fails to meet the target. Furthermore, the triggering conditions for iterative fine-tuning are based on quantifiable objective indicators (e.g., a comparison between the first ratio and the fourth threshold), eliminating the need for manual judgment and achieving automated decision-making in the TTS model enhancement process.

[0171] In addition, the processing unit 300 may be configured to obtain an adjusted TTS model in response to satisfying any of the following conditions, wherein the conditions include: the current ratio of the number of samples in the N+1th fine-tuning dataset to the number of samples in the N+1th candidate audio sample set is greater than or equal to a fourth threshold; the difference between the current ratio and the first ratio is less than a fifth threshold; or the number of fine-tuning operations performed so far reaches a preset maximum number of fine-tuning operations.

[0172] During the iterative fine-tuning process, after each round of fine-tuning, the yield rate corresponding to this round is calculated, or the ratio of the number of samples in the fine-tuning dataset to the number of samples in the candidate audio sample set. When this ratio reaches a preset fourth threshold, it indicates that the generation quality of the current TTS model meets the target requirements, and the iterative fine-tuning process can be terminated. Furthermore, to prevent the iterative fine-tuning process from failing to converge due to an excessively high fourth threshold, a convergence threshold condition is also set. If the yield rate improvement between two adjacent rounds, or the difference between the current round's yield rate and the previous round's yield rate, is less than a preset fifth threshold, it can be considered that the TTS model's capability improvement has reached saturation, and the benefits of continuing iterative fine-tuning are limited, at which point the iteration is also terminated. In addition, the implementation method of this disclosure also sets a maximum number of fine-tuning attempts as a safety mechanism to prevent the iterative fine-tuning process from entering an infinite loop due to abnormal situations that cause the first two conditions to be unmet.

[0173] By setting an absolute threshold condition (fourth threshold) during the iterative fine-tuning process, a clear quality objective is provided for iteration termination, ensuring that the final output TTS model has sufficient high-quality sample generation capability. Setting a convergence threshold condition (fifth threshold) avoids a large number of invalid iterations due to slow quality improvement, saving computational resources and ensuring that the TTS model terminates promptly when performance saturates. Setting a preset maximum number of fine-tuning iterations as a safety mechanism prevents infinite loops caused by unreasonable threshold settings or data anomalies, ensuring the robustness and controllability of the entire adjustment process. Therefore, the strategy of "terminating upon meeting any condition" balances quality objectives, efficiency assurance, and safety, achieving automation of the iterative fine-tuning process and adaptive termination decisions without manual intervention.

[0174] In some embodiments of this disclosure, the processing unit 300 may also be configured to: generate a candidate audio sample set (N+1) by random sampling based on the TTS model that has undergone the Nth fine-tuning; select the N+1th fine-tuning dataset from the candidate audio sample set based on the evaluation results of the multi-dimensional evaluation; and perform the N+1th fine-tuning on the TTS model that has undergone the Nth fine-tuning based on the seed dataset, the first fine-tuning dataset, and the fine-tuning dataset generated in the iterative fine-tuning.

[0175] For example, in some embodiments of this disclosure, the processing unit 300 may also be configured to: in response to the failure to meet any of the above conditions, generate a third candidate audio sample set according to the generation instructions in a random sampling manner based on the TTS model that has completed the second fine-tuning; select a third fine-tuning dataset from the third candidate audio sample set based on the evaluation results of the multi-dimensional evaluation; and perform a third fine-tuning on the TTS model that has completed the second fine-tuning based on the seed dataset, the first fine-tuning dataset, the second fine-tuning dataset, and the third fine-tuning dataset.

[0176] In this way, by automatically entering the next iteration when the termination condition is not met, the TTS model can continuously optimize itself using newly added high-quality samples until its performance reaches the target or converges, thus achieving a gradual improvement in capabilities. Each iteration incorporates all historically accumulated high-quality samples into the training data, ensuring that the TTS model does not forget the high-quality generation patterns it has mastered during the learning process and continuously absorbs new high-quality variants, which helps to improve the stability and generalization ability of the TTS model.

[0177] Based on this, the N+1th candidate audio sample set is generated by random sampling based on the TTS model that has been fine-tuned for the Nth time. The N+1th candidate audio sample set is evaluated in multiple dimensions to select the N+1th fine-tuning dataset. Based on the seed dataset, the first fine-tuning dataset, and the cumulative results from the second fine-tuning dataset to the N+1th fine-tuning dataset, the N+1th fine-tuning is performed on the TTS model that has been fine-tuned for the Nth time.

[0178] By using the recursive method described above, the mechanism of the second round of iteration can be extended to any subsequent round, achieving adaptive continuation and termination of the iteration process. This approach is logically consistent and easy to implement. Furthermore, each round accumulates all historical high-quality samples for fine-tuning, ensuring that the TTS model does not forget previously learned generation patterns while continuously improving.

[0179] In some embodiments of this disclosure, the processing unit 300 may also be configured to fine-tune the TTS model based on a first fine-tuning dataset and a general speech dataset, wherein the general speech dataset is a training sample set collected from a multi-category speech data source, and the multi-category speech data source includes at least one of a Mandarin multi-speaker speech data source, a multi-dialect general speech data source, and a multi-language basic speech data source.

[0180] A general speech dataset can include basic speech samples from different speakers, accents, and languages. These samples typically do not include specific target speech categories, such as paralinguistics, specific accents, or specific styles, but can help TTS models maintain or enhance their general speech synthesis capabilities.

[0181] During fine-tuning, the first fine-tuning dataset can be mixed with a general speech dataset. This allows the TTS model to learn specific category generation capabilities while avoiding forgetting or degrading its original basic speech synthesis capabilities.

[0182] It should be noted that the fine-tuning method based on the general speech dataset described above is not limited to the first fine-tuning. In the embodiments of this disclosure, this method can be used in any fine-tuning process during iterative fine-tuning, or in other words, the TTS model can be fine-tuned based on a combination of the currently accumulated fine-tuning dataset (e.g., seed dataset, first fine-tuning dataset, second fine-tuning dataset, etc.) and the general speech dataset. Whether to use this method and the proportion of use can be determined according to the actual needs of the scenario.

[0183] Furthermore, some embodiments of this disclosure provide a speech synthesis apparatus including a TTS model configured to generate speech based on input text to be synthesized, wherein the TTS model is obtained by adjusting the text-to-speech model adjustment method 1000 described in any of the above embodiments.

[0184] refer to Figures 1 to 7 As described above, in the text-to-speech model adjustment method 1000, a first candidate audio sample set is generated based on the initial TTS model using random sampling. Due to the diversity and randomness of the random sampling method, it ensures that a certain proportion of high-quality audio samples are present in the batch of audio samples generated by the initial TTS model (e.g., the first candidate audio sample set). By performing multi-dimensional evaluation on the first candidate audio sample set based on at least two of the following: speech category, timbre, naturalness, prosody, and audio quality, high-quality audio samples (e.g., the first fine-tuning dataset) can be selected from the batch of audio samples. This achieves automated amplification and quality screening of the fine-tuning data used for the initial TTS model, and improves the efficiency of data acquisition and the objectivity of the screening. When the number of audio samples used for training or fine-tuning is small, a sufficient number of high-quality audio samples can be obtained efficiently without increasing the training or fine-tuning cost of the initial TTS model, meeting the requirements for training or fine-tuning the initial TTS model.

[0185] After adjusting the initial TTS model using the text-to-speech model adjustment method 1000 described in any of the above embodiments, the obtained TTS model can be applied in speech synthesis scenarios where training or fine-tuning data is scarce. For example, in scenarios such as generating paralinguistics, specific accent speech, or specific style speech, the text to be synthesized can be input into the adjusted TTS model to generate synthesized speech with high semantic matching degree, high timbre consistency, and high audio quality.

[0186] Since the TTS model included in this speech synthesis device is obtained through the text-to-speech model adjustment method 1000, the aforementioned speech synthesis device naturally also possesses the various advantages brought by the text-to-speech model adjustment method 1000. For example, it can achieve stable generation of high-quality speech under conditions of scarce training or fine-tuning data, avoid problems such as category confusion, timbre drift, and quality degradation, and the entire synthesis process does not require the introduction of additional screening or correction steps, maintaining a simple and efficient inference flow. Other beneficial effects of the speech synthesis device will not be elaborated here.

[0187] According to embodiments of this disclosure, this disclosure also provides an electronic device, a non-volatile computer-readable storage medium, and a computer program product.

[0188] Figure 9 This is a schematic block diagram of an electronic device 3000 provided according to an exemplary embodiment of the present disclosure. The electronic device 3000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0189] like Figure 9 As shown, the electronic device 3000 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 3000. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0190] Multiple components in electronic device 3000 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 3000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0191] The computing unit 301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit, a graphics processing unit, various special-purpose artificial intelligence computing chips, various computing units running machine learning model algorithms, a digital signal processor, and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various means and processes described above, such as text-to-speech model adjustment methods or speech synthesis methods. For example, in some embodiments, the text-to-speech model adjustment methods and speech synthesis methods can be implemented as computer software programs, which are tangibly included in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 3000 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the text-to-speech model adjustment methods or speech synthesis methods described above can be performed. Alternatively, in other embodiments, the computing unit 301 may be configured by any other suitable means (e.g., by means of firmware) to perform a text-to-speech model adjustment method or a speech synthesis method.

[0192] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), complex programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These embodiments can be implemented as one or more computer programs that can execute on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] Program code for implementing the apparatus of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing method, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0194] In the context of this disclosure, a machine-readable medium can be a tangible medium that may include or store a program for use by or in conjunction with an instruction execution system, method, or apparatus. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, method, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display method for showing information to the user (e.g., a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display) monitor); and a keyboard and pointing method (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of methods can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0196] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0197] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0198] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, apparatuses, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0199] Various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements of the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0200] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0201] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.< / cough> < / laugh> < / cough> < / laugh> < / cough> < / laugh>

Claims

1. A method for adjusting a text-to-speech model, comprising: Based on the text-to-speech (TTS) model, a first candidate audio sample set is generated using random sampling. The first candidate audio sample set is evaluated in multiple dimensions to select the first fine-tuning dataset from the first candidate audio sample set; Based at least on the first fine-tuning dataset, the TTS model is fine-tuned to obtain the adjusted TTS model. The multi-dimensional evaluation includes evaluating at least two of the following: speech category, timbre, naturalness, prosody, and audio quality of the samples in the first candidate audio sample set.

2. The method according to claim 1, wherein, The method further includes: Before the step of generating the first candidate audio sample set by random sampling, the TTS model is trained based on the seed dataset to train the TTS model's ability to generate corresponding audio based on the input speech category label; The seed dataset includes audio samples and corresponding speech category labels, which are used to indicate the target speech category.

3. The method according to claim 2, wherein, The voice category labels include: At least one of the following: emotional expression tags for speech and physiological response tags for speech. The emotional expression tags of the speech include at least one of the following: laughter tag, crying tag, sighing tag, and exclamation tag; The physiological response tags for the speech include at least one of the following: cough tag, sneeze tag, yawn tag, and hiccup tag.

4. The method according to claim 3, wherein, The text-to-speech (TTS) model generates a first candidate audio sample set using random sampling, including: Based on the speech category label and the reference audio of the target speaker, a generation instruction is constructed, which instructs the TTS model to generate audio samples that match the speech category label and the timbre features of the target speaker in a random sampling manner; The generation instructions are input into the TTS model to obtain the first candidate audio sample set generated by the TTS model.

5. The method according to claim 4, wherein, The multi-dimensional evaluation of the first candidate audio sample set includes at least two of the following: The semantic matching degree of the first candidate audio sample set is evaluated to assess the consistency between the speech category expressed by the samples in the first candidate audio sample set and the speech category label, and a semantic matching metric value is generated. A timbre consistency evaluation is performed on the first candidate audio sample set to assess the consistency between the timbre of the samples in the first candidate audio sample set and the timbre of the reference audio of the target speaker, and a timbre consistency quantification value is generated. An audio quality assessment is performed on the first candidate audio sample set to evaluate whether there are any technical defects in the samples in the first candidate audio sample set, and an audio quality quantification value is generated.

6. The method according to claim 5, wherein, The step of performing a multi-dimensional evaluation of the first candidate audio sample set to select the first fine-tuning dataset from the first candidate audio sample set includes: The sample is included in the first fine-tuning dataset if the semantic matching quantization value of the sample is greater than or equal to a first threshold, the timbre consistency quantization value is greater than or equal to a second threshold, and the audio quality quantization value is greater than or equal to a third threshold.

7. The method according to claim 1, wherein, The step of performing a multi-dimensional evaluation of the first candidate audio sample set to select the first fine-tuning dataset from the first candidate audio sample set includes: The trained screening model is used to perform multi-dimensional evaluation on the first candidate audio sample set in order to select the first fine-tuned dataset from the first candidate audio sample set. The screening model includes a multimodal model.

8. The method according to claim 2, wherein, The fine-tuning of the TTS model based at least on the first fine-tuning dataset includes: Based on the seed dataset and the first fine-tuning dataset, the TTS model is subjected to its first fine-tuning. The adjusted TTS model is obtained when the first ratio of the number of samples in the first fine-tuned dataset to the number of samples in the first candidate audio sample set is greater than or equal to a fourth threshold.

9. The method according to claim 8, wherein, The method further includes: In response to the first ratio being less than the fourth threshold, at least one round of iterative fine-tuning is performed on the TTS model.

10. The method according to claim 9, wherein, The step of performing at least one round of iterative fine-tuning on the TTS model includes: Based on the TTS model that has undergone the Nth fine-tuning, the N+1th candidate audio sample set is generated using the random sampling method, where N is a positive integer greater than or equal to 1; The multi-dimensional evaluation is performed on the N+1th candidate audio sample set to select the N+1th fine-tuning dataset from the N+1th candidate audio sample set; Based on the seed dataset, the first fine-tuning dataset, and the fine-tuning dataset generated in the iterative fine-tuning, the TTS model that has completed the Nth fine-tuning is subjected to the N+1th fine-tuning.

11. The method according to claim 10, wherein, The step of performing at least one round of iterative fine-tuning on the TTS model also includes: The at least one round of iterative fine-tuning ends and the adjusted TTS model is obtained in response to any of the following conditions being met. The conditions include at least one of the following: The current ratio of the number of samples in the (N+1)th fine-tuning dataset to the number of samples in the (N+1)th candidate audio sample set is greater than or equal to the fourth threshold. The difference between the current ratio and the first ratio is less than the fifth threshold; The number of fine-tuning operations performed so far is greater than or equal to the preset maximum number of fine-tuning operations.

12. The method according to any one of claims 1 to 11, wherein, The samples in the first candidate audio sample set and the first fine-tuning dataset include sub-language audio samples, where the sub-language is a non-lexical semantic pronunciation.

13. The method according to claim 12, wherein, The fine-tuning of the TTS model based at least on the first fine-tuning dataset includes: The TTS model is fine-tuned based on the first fine-tuning dataset and the general speech dataset. The general speech dataset is a training sample set collected from multi-category speech data sources, which include at least one of the following: a Mandarin multi-speaker speech data source, a multi-dialect general speech data source, and a multi-language basic speech data source.

14. A speech synthesis method, comprising: The text to be synthesized is input into the TTS model to generate speech. The TTS model is obtained by adjusting it based on the adjustment method described in any one of claims 1 to 13.

15. An adjustment device for a text-to-speech model, comprising: The generation unit is configured to generate a first candidate audio sample set based on a text-to-speech (TTS) model using random sampling. The filtering unit is configured to perform multi-dimensional evaluation on the first candidate audio sample set in order to filter out the first fine-tuning dataset from the first candidate audio sample set. The processing unit, based at least on the first fine-tuning dataset, fine-tunes the TTS model to obtain an adjusted TTS model. The multi-dimensional evaluation includes evaluating at least two of the following: speech category, timbre, naturalness, prosody, and audio quality of the samples in the first candidate audio sample set.

16. A speech synthesis device, comprising: The TTS model is configured to generate speech based on the input text to be synthesized. The TTS model is obtained by adjusting it based on the adjustment method described in any one of claims 1 to 13.

17. An electronic device comprising: A processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the adjustment method of any one of claims 1 to 13 and the speech synthesis method of claim 14.

18. A non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the adjustment method of any one of claims 1 to 13 and the speech synthesis method of claim 14.

19. A computer program product comprising a computer program that, when executed by a processor, implements the adjustment method of any one of claims 1 to 13 and the speech synthesis method of claim 14.

Images