Model training method and apparatus, and electronic device

Abstract

This application discloses a model training method, apparatus, and electronic device, belonging to the field of artificial intelligence. The method includes: acquiring cloned audio samples generated by a sound cloning model using original corpus; identifying the cloned text corresponding to the cloned audio samples, comparing the cloned text with the original corpus, and filtering out qualified and unqualified samples; evaluating the qualified samples using multiple evaluation models through ensemble learning, obtaining the evaluation results independently output by each evaluation model, and constructing a training dataset through voting; performing distillation training on a sound synthesis model using the training dataset to obtain a trained sound synthesis model; evaluating the synthesized audio output by the trained sound synthesis model using multiple evaluation models through ensemble learning, and continuing distillation training if the evaluation results do not meet the conditions until the evaluation results of the synthesized audio meet the conditions. This application improves the quality and efficiency of model training.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and in particular relates to a model training method, apparatus and electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, text-to-speech (TTS) technology has been widely used in many fields. However, mainstream speech synthesis models usually require a large amount of data and high training costs. To address this, model distillation is a common technique to simplify speech synthesis models and reduce the high demand for computing resources and data storage. However, model distillation usually requires manual intervention in cleaning and evaluating training data, which is not only costly but also prone to poor data cleaning quality due to human error. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the related art. To this end, this application proposes a model training method, apparatus, and electronic device, which improves the automation level of training data selection during the model training process and improves the quality and efficiency of model training.

[0004] Firstly, this application provides a model training method, which includes: Obtain cloned audio samples generated from the original corpus using a sound cloning model, wherein the sound cloning model is trained based on a reference text and the reference audio corresponding to the reference text. The cloned text corresponding to the cloned audio sample is identified, and the cloned text is compared with the original corpus to filter the cloned audio samples to obtain qualified and unqualified samples; Based on the reference audio, the qualified samples are evaluated by multiple evaluation models through ensemble learning, and the evaluation results output independently by each evaluation model are obtained. The qualified samples are then screened by voting to obtain target samples to construct a training dataset. The sound synthesis model is distilled and trained using the training dataset to obtain the trained sound synthesis model. Based on the reference audio, the synthesized audio output by the trained sound synthesis model is evaluated using multiple evaluation models learned through ensemble learning. If the evaluation result of the synthesized audio does not meet the conditions, the process returns to the step of distilling the sound synthesis model using the training dataset and continues until the evaluation result of the synthesized audio meets the conditions.

[0005] In the above technical solution, a voice cloning model that has learned the voiceprint features of a specific user is used to generate cloned audio samples. Qualified cloned audio samples are selected by comparing the cloned text with the original corpus. Multiple evaluation models using ensemble learning are then used to evaluate the qualified samples. A voting mechanism is used to select target samples to construct the training dataset. Data cleaning during the distillation process can be performed without human intervention, improving the inconsistent quality of cleaning results caused by human error, increasing automation, and helping to improve the quality and efficiency of training data cleaning and verification. This provides a high-quality data foundation for subsequent voice synthesis model training, thereby improving the quality and efficiency of model training. When training the voice synthesis model, the knowledge of the voice cloning model is learned through distillation training, thereby reducing the model's complexity and computational requirements. Multiple evaluation models using ensemble learning evaluate the synthesized audio of the voice synthesis model, enabling it to efficiently replicate the voiceprint features of a specific user. Simultaneously, the model is lightweight, improving its applicability to real-time, high-concurrency text-to-speech applications.

[0006] According to one embodiment of this application, the step of evaluating the qualified samples based on the reference audio using multiple evaluation models of ensemble learning, and obtaining evaluation results independently output by each evaluation model, includes: The qualified samples are evaluated using the first evaluation model based on the reference audio, and the evaluation results are obtained. The qualified samples are evaluated using the second evaluation model to obtain the evaluation results; The qualified samples are evaluated using a third evaluation model based on their short-time energy (STE) and zero crossover rate (ZCR) characteristics, resulting in an evaluation outcome.

[0007] In the above technical solution, qualified samples are evaluated based on reference audio using a first evaluation model, qualified samples are evaluated using a second evaluation model, and qualified samples are evaluated based on the short-time energy (STE) feature and zero cross-rate (ZCR) feature of qualified samples using a third evaluation model. By integrating the evaluation results of multiple models through ensemble learning, it is helpful to capture complex patterns and relationships in audio data. Integrating multiple evaluation models can provide a more flexible evaluation strategy to adapt to different evaluation needs. By integrating the evaluation results of different models, the evaluation of qualified samples becomes more comprehensive and accurate.

[0008] Secondly, this application provides a model training apparatus, which includes: A generation module is used to obtain cloned audio samples generated by a sound cloning model using the original corpus. The sound cloning model is trained based on a reference text and the reference audio corresponding to the reference text. The filtering module is used to identify the cloned text corresponding to the cloned audio sample and compare the cloned text with the original corpus to filter the cloned audio samples to obtain qualified samples and unqualified samples. The evaluation module is used to evaluate the qualified samples based on the reference audio by using multiple evaluation models learned through ensemble learning, obtain the evaluation results output independently by each evaluation model, and filter the qualified samples by voting to obtain target samples to construct a training dataset. The training module is used to perform distillation training on the sound synthesis model using the training dataset to obtain the trained sound synthesis model. The evaluation module is further configured to evaluate the synthesized audio output by the trained sound synthesis model based on the reference audio and through multiple evaluation models learned through ensemble learning. If the evaluation result of the synthesized audio does not meet the conditions, the module returns to the step of distilling the sound synthesis model using the training dataset and continues execution until the evaluation result of the synthesized audio meets the conditions.

[0009] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method as described in the first aspect above.

[0010] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training method as described in the first aspect above.

[0011] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the model training method as described in the first aspect.

[0012] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the model training method as described in the first aspect above.

[0013] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0014] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating the model training method provided in some embodiments of this application; Figure 2 This is a second schematic flowchart of a model training method provided in some embodiments of this application; Figure 3 This is the third of the flowcharts illustrating the model training method provided in some embodiments of this application; Figure 4 This is a schematic diagram of the structure of the second evaluation model provided in some embodiments of this application; Figure 5 This is the fourth of the flowcharts illustrating the model training method provided in some embodiments of this application; Figure 6 This is a schematic diagram of the structure of a model training device provided in some embodiments of this application; Figure 7 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0016] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0017] The model training method, apparatus, and electronic equipment provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0018] The model training method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the model training method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The model training method provided in this application embodiment will be described below using an electronic device as the execution subject.

[0019] Figure 1This is one of the flowcharts illustrating the model training method provided in some embodiments of this application. For example... Figure 1 As shown, the model training method includes steps 110, 120, 130, 140, and 150.

[0020] Step 110: Obtain cloned audio samples generated by the sound cloning model using the original corpus. The sound cloning model is trained based on the reference text and the reference audio corresponding to the reference text.

[0021] In this embodiment, the voice cloning model has been trained based on reference text and its corresponding reference audio. For example, the reference audio can be reading or speaking content recorded by a specific user based on the reference text. It can be considered that the voice cloning model has learned the voiceprint features of a specific user (such as timbre, pitch, and speaking speed habits) and can be used to replicate audio that matches those features. In some embodiments, the voice cloning model is a trained model, such as the Large Language Model-based (LLM-based VC) voice cloning model in the CosyVoice framework, and is fine-tuned based on a small amount of reference text and reference audio.

[0022] The original corpus can be text data that includes multiple sentences or phrases. For example, the original corpus can consist of texts of multiple Chinese sentences and English sentences. The cloned audio sample is audio data generated by the voice cloning model based on the original corpus, which has the voiceprint features (such as timbre, pitch, speech rate habits, etc.) of a specific user.

[0023] In this embodiment, the electronic device collects or creates the original corpus, providing input data for the sound cloning model. This ensures that the generated cloned audio samples are produced using the original corpus through the sound cloning model. This enables the sound cloning model to generate cloned audio samples that match the original corpus. After evaluation, the cloned audio samples can be used for distillation training of the sound synthesis model.

[0024] Step 120: Identify the cloned text corresponding to the cloned audio sample, and compare the cloned text with the original corpus to screen the cloned audio samples to obtain qualified and unqualified samples.

[0025] Identifying the cloned text corresponding to the cloned audio sample can be achieved, for example, through speech recognition technology, such as rule-based speech recognition or statistical model-based speech recognition, to recognize the cloned audio sample as its corresponding text. After identifying the corresponding cloned text, the cloned text is compared with the original corpus. It is understood that the comparison result can be used to evaluate the performance of the audio cloning model. If the current cloned text matches the original corpus, it indicates that the model's performance is good, and the cloned audio sample corresponding to the current cloned text can be considered a qualified sample. If the current cloned text contains errors or inconsistencies compared to the original corpus, such as pronunciation or grammatical errors, the cloned audio sample corresponding to the current cloned text is considered an unqualified sample. In some embodiments, the inconsistencies between the cloned text and the original corpus can be used to guide further optimization and adjustment of the sound cloning model.

[0026] By comparing cloned text with the original corpus, the accuracy and usability of cloned audio samples are verified. Cloned audio samples corresponding to cloned text with consistent comparison results are considered qualified samples. Qualified samples provide a data foundation for subsequent audio processing steps (such as audio evaluation and audio screening). Selecting high-quality cloned text samples during construction to build the training dataset helps improve the training effect of distillation training of the sound synthesis model using the training dataset.

[0027] The following is a specific example of identifying the cloned text corresponding to a cloned audio sample. and clone the text Compared with the original corpus To make a comparison, that is, to judge the text Compared with the original corpus Are they consistent? :

[0028] in , ; Understandably, determining cloned text Compared with the original corpus The method to determine consistency is to check the cloned text separately. and the original corpus All characters except punctuation and spaces are lowercase. In cloned text Compared with the original corpus If the above processing results in consistency, the cloned audio sample corresponding to the cloned text is determined to be a qualified sample. Compared with the original corpus If there is no consistency after the above processing, the cloned audio sample corresponding to the cloned text is determined to be an unqualified sample.

[0029] Step 130: Based on the reference audio, the qualified samples are evaluated by multiple evaluation models through ensemble learning to obtain the evaluation results output independently by each evaluation model, and the qualified samples are screened by voting to obtain target samples to construct a training dataset.

[0030] As is understandable, ensemble learning refers to using multiple learners to solve the same problem and integrating the results of these learners to obtain the final result. These learners are usually poor-performing models, but by ensemble, they can be combined into a more powerful model, which helps to reduce overfitting or underfitting and improve the overall performance of the model.

[0031] In this embodiment, evaluating qualified samples using multiple evaluation models through ensemble learning refers to improving overall evaluation performance by combining multiple evaluation models and integrating the independent evaluation results of each model to obtain the final evaluation result. Specifically, different evaluation models may perform better on different data audio sample features. By integrating these models, problems such as evaluation errors caused by a single model can be reduced, which helps to improve the accuracy and robustness of evaluating qualified samples.

[0032] Understandably, multiple evaluation models refer to a series of different audio evaluation models, each potentially focusing on evaluating different aspects of the audio, such as voiceprint features, sound wave signals, etc. These evaluation models can be built based on different algorithms or techniques, such as neural networks, support vector machines, decision trees, etc., to capture different features of the audio signal. Each evaluation model independently evaluates the same audio sample (pass) and outputs its evaluation result, which may be in the form of a score, grade, or pass / fail judgment.

[0033] In ensemble learning, voting mechanisms are typically used to integrate the evaluation results of multiple models. For example, if a majority of the evaluation models consider a given sample to have passed, then that sample is considered to have passed. The voting mechanism can be a simple majority vote or a weighted voting system, where each evaluation model's result is assigned different weights based on its performance or importance. Through ensemble learning, multiple evaluation models are used to select qualified samples that pass the evaluation; these samples are then used to construct the training dataset.

[0034] Understandably, the voice cloning model has been trained based on reference text and its corresponding reference audio, learning the voiceprint features of a specific user and capable of replicating audio that matches those features. Since the reference audio typically contains the voiceprint features of a specific user, the similarity between qualified samples and the reference audio in voiceprint features can be evaluated based on the reference audio. From the qualified samples, those that pass the evaluation (i.e., are consistent with the reference audio in voiceprint features) are selected as target samples. Further training datasets are constructed based on these target samples to perform distillation training on the voice synthesis model. This ensures that the training data used for the voice synthesis model consists entirely of qualified samples consistent with the reference audio in voiceprint features, improving the effectiveness of distillation training and enabling the voice synthesis model to better learn the voice cloning model's ability to replicate the voiceprint features of a specific user. Data cleaning during the distillation process can be performed without human intervention, which improves the problem of inconsistent cleaning results caused by human error. Compared with related technologies that require manual intervention to clean and verify training data during distillation training, this method improves the degree of automation and helps to improve the quality and efficiency of training data cleaning and verification (corresponding to the evaluation and screening of qualified samples in this application).

[0035] Step 140: Use the training dataset to perform distillation training on the sound synthesis model to obtain the trained sound synthesis model.

[0036] Understandably, the voice synthesis model should be a lightweight model compared to the voice cloning model. Therefore, knowledge transfer from the voice cloning model to the voice synthesis model can be achieved through model distillation training, reducing model complexity and computational requirements. For example, the voice synthesis model could be an end-to-end text-to-speech (TTS) model, such as a model built on the TacoTron2 architecture combined with vocoders like HiFi-GAN or Parallel WaveGAN. Its initial weights could be a pre-trained general TTS model, which learns the ability to generate voiceprint features specific to a user from the voice cloning model through distillation training.

[0037] Figure 2 This is the second schematic flowchart of a model training method provided in some embodiments of this application. For example... Figure 2 As shown, the voice cloning model has been trained based on user recording data (corresponding to the reference text and its corresponding reference audio), and the voice synthesis model has been trained based on the data generated by the voice cloning model. The data generated by the voice cloning model corresponds to the training dataset constructed earlier by filtering qualified samples to obtain target samples.

[0038] Step 150: Based on the reference audio, evaluate the synthesized audio output by the trained sound synthesis model using multiple evaluation models through ensemble learning. If the evaluation result of the synthesized audio does not meet the conditions, return to the step of distilling the sound synthesis model using the training dataset and continue execution until the evaluation result of the synthesized audio meets the conditions.

[0039] Figure 3 This is the third flowchart illustrating the model training method provided in some embodiments of this application. Figure 3 As shown, after the voice synthesis model is trained, the synthesized audio output by the trained voice synthesis model can be evaluated using multiple evaluation models through ensemble learning. This evaluation method is the same as that used to evaluate qualified samples. If the evaluation result does not meet the conditions, such as failing, the voice synthesis model is retrained using the training dataset until the conditions are met, such as passing the evaluation. This allows the voice synthesis model to capture the voiceprint features of a specific user and reproduce audio that matches those features.

[0040] Understandably, voice cloning models typically have good voice cloning effects, achieving high-quality voice cloning and synthesis. However, they usually require massive computational resources and are difficult to apply to real-time, high-concurrency text-to-speech scenarios. In contrast, voice synthesis models are usually more lightweight than voice cloning models. Furthermore, by distilling and training based on the generated data of selected voice cloning models, they achieve the ability to learn the voiceprint features of specific users while reducing the demand for storage and computing resources, thus improving adaptability to high-real-time application scenarios.

[0041] The model training method provided in this application utilizes a voice cloning model that has learned the voiceprint features of a specific user to generate cloned audio samples. Qualified cloned audio samples are selected by comparing the cloned text with the original corpus. Multiple evaluation models using ensemble learning are then used to evaluate the qualified samples. A voting mechanism is used to select target samples to construct the training dataset. This method allows for data cleaning during the distillation process without human intervention, improving the inconsistent quality of cleaning results caused by human error, increasing automation, and enhancing the quality and efficiency of training data cleaning and verification. This provides a high-quality data foundation for subsequent voice synthesis model training, thereby improving the quality and efficiency of model training. When training the voice synthesis model, the knowledge of the voice cloning model is learned through distillation training, reducing the model's complexity and computational requirements. Multiple evaluation models using ensemble learning evaluate the synthesized audio of the voice synthesis model, enabling it to efficiently replicate the voiceprint features of a specific user. Simultaneously, the model is lightweight, improving its applicability to real-time, high-concurrency text-to-speech applications.

[0042] In some embodiments of this application, the step of evaluating the qualified samples based on the reference audio using multiple evaluation models ensembled through learning, and obtaining evaluation results independently output by each evaluation model, includes: The qualified samples are evaluated using the first evaluation model based on the reference audio, and the evaluation results are obtained. The qualified samples are evaluated using the second evaluation model to obtain the evaluation results; The qualified samples are evaluated using a third evaluation model based on their short-time energy (STE) and zero crossover rate (ZCR) characteristics, resulting in an evaluation outcome.

[0043] This can be understood as the first, second, and third evaluation models corresponding to three different methods for evaluating qualified samples. For example, the first evaluation model could be an implicit distribution evaluation model of sound features based on T-distributed stochastic neighbor embedding (T-SNE), the second evaluation model could be a speech quality assessment (SQA) model based on the Mamba structure, and the third evaluation model could be an evaluation model based on sound wave signals, and so on.

[0044] For example, the first evaluation model can be used to compare the feature values ​​of qualified samples with those of reference audio to evaluate qualified samples. By extracting features from both qualified samples and reference audio in the same manner, the similarity between the qualified samples and reference audio in terms of voiceprint or acoustic features is quantified, and qualified samples that are consistent with the reference audio in terms of timbre, pitch, etc., are determined to pass the evaluation. For example, this could involve extracting Mel-frequency cepstral coefficients (MFCC) feature values ​​from both qualified samples and reference audio, or it could involve extracting time-frequency features or time-series characteristics, etc.

[0045] For example, the second evaluation model can assess qualified samples based on human auditory perception, providing scores that simulate human evaluation of audio quality, such as the Mean Opinion Score (MOS). By inputting qualified samples into a pre-trained second evaluation model, which may include multi-layer neural networks or other machine learning algorithms, the model outputs a MOS. The evaluation results obtained through the second evaluation model help capture the subjective quality of qualified samples, such as naturalness and clarity.

[0046] For example, the third evaluation model can evaluate qualified samples based on their physical characteristics, such as Short-Time Energy (STE) and Zero Crossing Rate (ZCR). The STE reflects the energy change of the audio signal of a qualified sample within a short time window, while the ZCR reflects the frequency at which the audio signal of a qualified sample crosses zero. Further evaluation of qualified samples based on the STE and ZCR characteristics allows for the identification of noise or other interference factors that may affect audio quality by analyzing the acoustic signal characteristics of qualified samples.

[0047] Understandably, different evaluation models may be sensitive to different features of audio. Ensemble learning leverages the diversity of multiple models by combining their evaluation results. Each model can analyze data from different perspectives, and this diversity helps capture complex patterns and relationships within the data. It also improves the problem of single models being overly sensitive to certain types of features or noise. Furthermore, integrating multiple models enhances their generalization ability to unknown data, and different application scenarios may have different requirements for audio quality. Integrating multiple evaluation models can provide more flexible evaluation strategies to adapt to different application needs. Moreover, when multiple independent models give similar evaluation results for the same audio sample, it can increase confidence in those evaluation results.

[0048] The model training method provided in this application evaluates qualified samples based on reference audio using a first evaluation model, a second evaluation model, and a third evaluation model based on the short-time energy (STE) and zero-crossing rate (ZCR) features of qualified samples. By integrating the evaluation results of multiple models through ensemble learning, it helps to capture complex patterns and relationships in audio data. Integrating multiple evaluation models can provide a more flexible evaluation strategy to adapt to different evaluation needs. By integrating the evaluation results of different models, the evaluation of qualified samples becomes more comprehensive and accurate.

[0049] In some embodiments of this application, the step of evaluating the qualified sample based on the reference audio using a first evaluation model to obtain an evaluation result includes: The MFCC feature values ​​of the qualified samples and the MFCC feature values ​​of the reference audio are obtained respectively. The MFCC feature values ​​of the qualified samples and the MFCC feature values ​​of the reference audio are reduced in dimensionality by using T-distributed random nearest neighbor embedding (T-SNE). Calculate the Mahalanobis distance between the MFCC eigenvalues ​​of the qualified samples after dimensionality reduction and the MFCC eigenvalues ​​of the reference audio. The evaluation result of the qualified sample is determined based on the Mahalanobis distance and the preset distance threshold.

[0050] Extracting MFCC features from qualified samples and reference audio typically involves framing the audio signal, applying window functions, calculating Fourier transforms, using Mel filter banks, and performing discrete cosine transforms. The Mel Frequency Cepstrum Coefficient (MFCC) features of qualified samples and reference audio can capture key acoustic properties of the audio, such as timbre, pitch, and formants.

[0051] The T-distributed stochastic neighbor embedding (T-SNE) algorithm is used to reduce the dimensionality of the MFCC feature values ​​of qualified samples and reference audio. T-SNE is a non-linear dimensionality reduction technique that preserves the relative distances between data points in the low-dimensional space, which is helpful for data visualization and comparison. Mapping high-dimensional MFCC feature values ​​to a low-dimensional space facilitates effective comparison and analysis.

[0052] Mahalanobis distance is an efficient method for calculating the similarity between two unknown sets of samples. Unlike Euclidean distance, it takes into account the relationships between various characteristics, calculating the Mahalanobis distance between the eigenvalues ​​of the qualified sample and the reference audio after dimensionality reduction. Specifically, it involves calculating the Euclidean distance between the mean vectors of the two samples, while also considering the covariance matrix of the samples.

[0053] Based on the comparison between the Mahalanobis distance and a preset threshold, it is determined whether a qualified sample is sufficiently similar to the reference audio in acoustic features. For example, if the calculated Mahalanobis distance is less than or equal to the preset distance threshold, the qualified sample is considered similar to the reference audio in acoustic features, and the evaluation result is passed; otherwise, the evaluation result is failed. In some embodiments, the preset distance threshold is set to 0.95.

[0054] Here is a specific example: The implicit distribution evaluation method for sound features based on T-SNE consists of (1) calculating the MFCC features of the audio; and (2) performing T-SNE implicit dimensionality reduction on the MFCC features. (1) Calculate the MFCC features of the audio:

[0055]

[0056]

[0057] in This represents the power spectral density of the audio signal at the k-th frequency point. This indicates the length of the Fast Fourier Transform (FFT), which is the number of sampling points in the analysis window. This represents the amplitude of the audio signal at the k-th frequency point. This represents the signal energy after passing through the m-th Mel filter. This represents the response of the m-th Mel filter at the k-th frequency point. This represents the nth Mel-frequency cepstral coefficient. It represents the number of Mel filters.

[0058] (2) Perform T-SNE implicit dimensionality reduction on MFCC features: Let the reference audio be A, and the characteristics of the reference audio A are: :

[0059] The set of qualified samples is B, and the corresponding feature set is That is, the characteristics of the qualified sample set B. :

[0060] Where N is the number of qualified samples. Represents the D-dimensional real space; For reference audio A and qualified sample set B:

[0061] in This indicates that the reference audio A and the j-th qualified sample in the qualified sample set are... The similarity probability, where N represents the total number of samples in the qualified sample set. This represents the bandwidth parameter in the Gaussian kernel function, used to control the size of the local neighborhood.

[0062] Suppose that the features of the reference audio A obtained after performing T-SNE dimensionality reduction on the reference audio and the qualified sample are respectively: :

[0063] The feature set corresponding to the qualified sample set B is ,Right now

[0064] Then the mean can be obtained. for:

[0065] variance for:

[0066] in, This represents the average distance between all feature values ​​in the feature set corresponding to the qualified sample set B and the features of the reference audio A. This represents the average variance of the distances between all feature values ​​in the feature set corresponding to B and the features of the reference audio A.

[0067] The Mahalanobis distance is used to determine the inclusion relationship between the features of reference audio A and the feature set, specifically defined as: Let the mean vector of the feature set corresponding to the qualified sample set B be...

[0068] Covariance matrix of the feature set

[0069] Calculate the Mahalanobis distance between the features of reference audio A and the feature set. : like If the value is less than 0.95, it can be considered that the reference audio A falls within the distribution area of ​​the qualified sample set B.

[0070] The model training method provided in this application extracts Mel-frequency cepstral coefficients (MFCC) feature values ​​of qualified samples and reference audio through a first evaluation model. These feature values ​​can capture key acoustic properties of the audio, such as timbre, pitch, and formants. Subsequently, the high-dimensional MFCC features are nonlinearly reduced in dimensionality using the T-distributed random nearest neighbor embedding (T-SNE) algorithm to maintain the relative distance between data points in the low-dimensional space. In the dimensionality-reduced space, the Mahalanobis distance between the qualified sample and the reference audio is calculated. The calculated Mahalanobis distance is compared with a preset threshold to determine whether the qualified sample is sufficiently similar to the reference audio in acoustic features. This achieves automated evaluation of qualified samples based on reference audio, thereby improving the quality and efficiency of sample evaluation and screening.

[0071] In some embodiments of this application, the step of evaluating the qualified samples using a second evaluation model to obtain evaluation results includes: The qualified samples are input into a pre-trained second evaluation model, which outputs the average opinion score (MOS) of the qualified samples; the second evaluation model includes a multi-layer Mamba network. The evaluation result of the qualified sample is determined based on the average opinion score and the first scoring threshold.

[0072] Understandably, the second evaluation model is a pre-trained model comprising a multi-layered Mamba network. Mamba networks are well-suited for processing sequential data and can capture long-term dependencies, making them suitable for evaluating audio quality. For example, the second evaluation model is trained by learning from a large number of audio samples and their corresponding Mean Opinion Score (MOS), thereby predicting the MOS score for input samples (qualified samples). MOS is a widely used audio quality metric that simulates subjective human evaluation of audio quality, typically ranging from 1 to 5, with scores positively correlated with audio quality.

[0073] In some embodiments, if the MOS is less than a first scoring threshold, the evaluation result of a qualified sample is determined to be a failure, and if the MOS is greater than or equal to the first scoring threshold, the evaluation result of a qualified sample is determined to be a pass.

[0074] Figure 4 This is a structural diagram of the second evaluation model provided in some embodiments of this application. For example... Figure 4 As shown, the second evaluation model includes: The Mel-Spectrogram layer is used to convert the input audio signal (qualified sample) into a Mel spectrogram. The Linear Project layer is used to project the Mel spectrogram. The Mamba layer is used to further extract features, where the Mamba network structure is repeated N times to enhance the feature extraction capability. The Global Pooling layer is used to compress the spatial dimensions of the feature map; The MLP Head (Multilayer Perceptron Head) layer, typically composed of one or more fully connected layers, is used to map globally pooled features to the final output space. In this embodiment, the MLP Head layer maps features to a scalar value representing the MOS score.

[0075] The model training method provided in this application uses a pre-trained second evaluation model, which includes a multi-layer Mamba network. This model is suitable for processing audio data such as qualified samples, which helps improve the quality of sample evaluation. It can evaluate qualified samples without human intervention, realize the automated evaluation of the subjective quality of audio samples, improve the problem of inconsistent evaluation results caused by human error, and help improve the quality and efficiency of sample evaluation.

[0076] In some embodiments of this application, the evaluation of the qualified samples using a third evaluation model based on their short-time energy (STE) and zero-crossing rate (ZCR) characteristics to obtain evaluation results includes: The qualified sample is segmented into at least one audio frame according to the preset frame length, and the STE feature and ZCR feature of each audio frame are calculated. The noise score of the qualified sample is calculated based on the STE and ZCR features of each audio frame. The evaluation result of the qualified sample is determined based on the noise score and the second scoring threshold.

[0077] Short-Time Energy (STE) is a measure of the energy of an audio signal over a short period of time, typically used to analyze the dynamic changes of audio signals. STE characteristics reflect changes in the loudness and intensity of an audio signal and are an important indicator for evaluating audio quality. Zero Crossing Rate (ZCR) is the number of times an audio signal crosses zero (i.e., positive-to-negative transitions) per unit time. ZCR characteristics are commonly used to analyze the frequency components of an audio signal.

[0078] In this embodiment, qualified samples are segmented into audio frames, and the STE and ZCR features of each audio frame are calculated. For example, qualified samples are segmented into multiple audio frames according to a preset frame length (e.g., 20ms or 30ms). Then, the STE and ZCR features of each frame are calculated. Based on the calculated STE and ZCR features, a comprehensive noise score is calculated for the qualified samples. This score can be based on a specific algorithm or model, such as a machine learning model, to evaluate the noise level of the audio. It is understood that the noise score provides a quantitative indicator to evaluate the noise level of the audio, helping to identify and exclude audio samples with high noise levels, thereby improving audio quality. The calculated noise score is compared with a preset second scoring threshold to determine the evaluation result of the qualified sample. For example, if the noise score of a qualified sample is less than the threshold, the evaluation result of the qualified sample is considered to be passed; otherwise, further processing or exclusion is required.

[0079] Here is a specific example: Noise score The specific calculation method is as follows:

[0080] Where NoiseScore[n] represents the noise score of the nth audio frame; and These are weighting coefficients, used to adjust the weights of ZCR and STE features in the noise score, respectively. This represents the ZCR feature value of the nth audio frame; This represents the STE feature of the nth audio frame; Let represent the mean and standard deviation of the ZCR features of n audio frames, respectively. Let represent the mean and standard deviation of the STE features of n audio frames, respectively; STE features of the nth audio frame The calculation formula is:

[0081] in, The frame length can be 20ms; It is an index variable used to iterate over each sample point in the audio frame; For window functions; Indicates the first The audio signal value of each sampling point; ZCR feature value of the nth audio frame The calculation formula is:

[0082] in, For symbolic functions, specifically:

[0083] The model training method provided in this application evaluates the audio quality of qualified samples by analyzing the short-time energy (STE) and zero-crossing rate (ZCR) characteristics of qualified samples using a third evaluation model. Specifically, the audio samples are divided into multiple frames, and the STE and ZCR characteristics of each frame are calculated. The STE characteristics reflect the energy changes of the audio signal in a short time, while the ZCR characteristics measure the frequency at which the signal crosses zero. By combining the STE and ZCR characteristics and adjusting their relative importance in the scoring using weighting coefficients, a noise score is obtained. The calculated noise score is compared with a preset second scoring threshold to determine the evaluation result of the audio sample. This method enables the identification of noise or other interference factors that may affect audio quality by analyzing the acoustic signal characteristics of qualified samples. It allows for the evaluation of qualified samples without human intervention, which helps improve the quality and efficiency of sample evaluation.

[0084] In some embodiments of this application, the step of filtering the qualified samples through voting to obtain target samples for constructing a training dataset includes: If there are no multiple evaluation results that are not "fail" in the independent output evaluation results, the qualified sample is determined as the target sample.

[0085] Understandably, target samples refer to qualified samples that have been evaluated and screened and are deemed suitable for building a training dataset. These samples will be used to train the sound synthesis model to improve the model's performance and output quality.

[0086] Voting selection is a decision-making method in which multiple models or evaluators independently evaluate data, and then determine the final decision based on majority or specific rules. In the embodiments of this application, voting selection can synthesize the opinions of multiple evaluation models to improve the accuracy and reliability of the decision. Specifically, voting selection is performed based on the output results of each evaluation model. If a qualified sample passes all evaluations, or fails no more than once, then the sample is determined as the target sample. The voting selection mechanism ensures that only qualified samples that pass the majority of evaluations are selected, thereby improving the quality of the training dataset.

[0087] Understandably, qualified samples that pass the evaluation (i.e., those consistent with the reference audio in voiceprint features) are selected from the qualified samples and used as target samples. Further training datasets are constructed based on these target samples to conduct distillation training on the voice synthesis model. This ensures that the training data used for the voice synthesis model training are all qualified samples consistent with the reference audio in voiceprint features, which helps improve the effectiveness of distillation training on the voice synthesis model. This allows the voice synthesis model to better learn the ability of the voice cloning model to replicate the voiceprint features of a specific user.

[0088] The following is a specific example of a voting filter: The independent evaluation results of the multiple evaluation models of ensemble learning are B1, B2 and B3, respectively;

[0089] When B1, B2, or B3 is 0, it indicates that the corresponding evaluation result is "fail"; when B1, B2, or B3 is 1, it indicates that the corresponding evaluation result is "pass". The final evaluation result is as follows:

[0090] In other words, if there are no multiple "fail" evaluation results among the independent outputs, the qualified sample is determined as the target sample.

[0091] In some embodiments, the voting mechanism may also be a weighted voting mechanism, where the evaluation results of each evaluation model are assigned different weights based on their performance or importance.

[0092] The model training method provided in this application evaluates audio samples using multiple evaluation models through ensemble learning, and determines target samples to construct a training dataset through a voting screening mechanism. During the voting screening process, if a qualified sample passes all evaluations or fails no more than once, the sample is determined as a target sample. This ensures that only those samples that pass most evaluations are selected, thereby improving the quality of the training dataset. By selecting samples with high audio quality and similar voiceprint features to the reference audio as target samples through voting, the training dataset is subsequently constructed based on these target samples, and the voice synthesis model is distilled and trained. This helps improve the distillation training effect, enabling the voice synthesis model to better learn the ability of the voice cloning model to replicate the voiceprint features of a specific user.

[0093] In some embodiments of this application, the method further includes: If the cloned audio sample is unqualified, reconstructed text is generated based on the cloned text corresponding to the unqualified sample; Using the reconstructed text as the original corpus, the process returns to the step of obtaining cloned audio samples generated using the original corpus through the sound cloning model to continue execution.

[0094] Generating new text based on the text corresponding to unqualified cloned audio samples is called reconstructing text. This process may involve natural language processing techniques such as synonym replacement, sentence transformation, or semantic editing to change the structure or content of the text while preserving as much of the original semantics as possible. Reconstructed text refers to the new text generated based on the text corresponding to unqualified cloned audio samples. By generating reconstructed text and regenerating cloned audio samples, the quality of the audio samples can be improved.

[0095] Figure 5 This is the fourth flowchart illustrating the model training method provided in some embodiments of this application. Figure 5 As shown, when a cloned audio sample is evaluated as unqualified, reconstructed text is generated based on the corresponding text. The purpose of reconstructing the text is to improve the output quality of the sound cloning model by changing the way the text is expressed without altering the original semantics. Using the reconstructed text as new original corpus, new cloned audio samples are generated again through the sound cloning model. This process may be repeated multiple times until the generated audio samples meet the quality requirements.

[0096] By continuously iterating and optimizing the text content, the quality of the cloned audio samples can be gradually improved so that the final cloned audio samples meet specific standards. The newly generated cloned audio samples are then fed back into the evaluation model for evaluation. If they are still unsatisfactory, new reconstructed text is generated based on the text corresponding to the new unsatisfactory samples, and the above process is repeated. This cyclical execution mechanism ensures that only high-quality audio samples are used to construct the training dataset, thereby improving the overall quality of the training dataset.

[0097] The model training method provided in this application, when the cloned audio samples are unqualified, generates reconstructed text based on the cloned audio samples, and uses this text as raw material to regenerate cloned audio samples. Without changing the original semantics, it improves the quality of cloned audio samples by changing the expression of the text and continuously iterating and optimizing the text content until the cloned text obtained from the cloned audio samples can be consistent with the original corpus. This avoids low-quality cloned audio samples with semantic inconsistencies from entering the subsequent processing flow, effectively improving the quality of cloned audio samples. It automates the reconstruction of cloned text and the regeneration of cloned audio samples without human intervention, which helps to improve the efficiency and quality of sample screening.

[0098] In some embodiments of this application, generating reconstructed text based on the cloned text corresponding to the unqualified sample includes: The cloned text and the original corpus are segmented into words respectively to obtain the first segmented word sequence corresponding to the cloned text and the second segmented word sequence corresponding to the original corpus. When the parts of speech of each corresponding word in the first word segmentation sequence and the second word segmentation sequence are the same, the words in the first word segmentation sequence and the second word segmentation sequence are swapped to maximize the difference between the first word segmentation sequence and the swapped first word segmentation sequence. The reconstructed text is generated based on the first word segmentation sequence after the exchange.

[0099] Understandably, word segmentation is the process of dividing text into individual words, phrases, or other meaningful elements. Parts of speech, on the other hand, refer to the grammatical function of words in a sentence, such as nouns, verbs, and adjectives, which helps in understanding the usage and meaning of words in the text.

[0100] Word segmentation sequences refer to the sequence of words obtained after text has been segmented. They represent the structure of the text and are the foundation for text analysis and processing. For example, natural language processing tools (such as NLTK, spaCy, etc.) are used to segment the cloned text and the original corpus corresponding to the unqualified samples, resulting in two word segmentation sequences. The first word segmentation sequence is obtained by segmenting the cloned text, and the second word segmentation sequence is obtained by segmenting the original corpus.

[0101] In two sets of word segments, segments with the same part of speech are identified, and these segments are swapped to increase the dissimilarity between the two sequences. By swapping segments, text that differs structurally from the original text can be generated, which helps improve text diversity while preserving the original semantics. Dissimilarity refers to the degree of difference between two texts or text sequences; it is used to measure the degree of change between the reconstructed text and the original text.

[0102] Based on the swapped second word segment sequence, the reconstructed text is generated by recombination. The purpose of generating the reconstructed text is to obtain a new text that is structurally different from the original text but semantically similar. This new text will be used as a new input for the sound cloning model in order to generate higher quality cloned audio samples.

[0103] Here is a specific example: The set of cloned texts corresponding to the unqualified samples :

[0104] Among them, set Includes N cloned texts ; For each cloned text Independent word segmentation yields the corresponding word segmentation sequence. :

[0105] in, Represents word segmentation sequence The k-th word in the middle, The corresponding part-of-speech tag is defined as It should be noted that word segmentation must ensure the smallest syntactic unit (such as noun phrases or verb phrases) and avoid breaking down the core semantics (e.g., "NewYork" is not segmented). Output the set of clone texts corresponding to the non-compliant samples. The corresponding set of segmented word sequences :

[0106] The text content exchange process is designed based on the degree of difference, where the degree of difference is... Defined as:

[0107] in, To reconstruct the word segmentation sequence of the pre-cloned text, Indicates the reconstructed word segmentation sequence; if A value of 0 indicates a word segmentation sequence. With word segmentation sequence exactly the same, if A value of 1 indicates a word segmentation sequence. With word segmentation sequence No two elements are the same. Positively correlated with the degree of difference; Let the word segmentation sequence be set and The corresponding part-of-speech sequences are as follows: and Word segmentation sequence and Word segmentation swapping between them must satisfy the following:

[0108] Where i represents the i-th word segmentation. Represents word segmentation sequence The part of speech corresponding to the i-th word in the text. Represents word segmentation sequence The part of speech corresponding to the i-th word.

[0109] In some embodiments, the word segmentation sequence of the cloned text cannot meet the conditions. In this case, adjustments are made by expanding the elements in the full word sequence to satisfy the condition.

[0110] Maximize the difference between the first word segmentation sequence and the swapped first word segmentation sequence, that is:

[0111] When swapping the first word segment sequence, the swapped first word segment sequence ( ) and the original first word segmentation sequence ( The parts-of-speech sequences should be as different as possible, that is, the parts-of-speech sequences of each word segment should remain unchanged (i.e., ... Under the premise of ), the degree of difference between the reconstructed sentence and the original sentence is increased. The goal of this process is to preserve the consistency of part-of-speech tags in the first word segmentation sequence, while maximizing... The differences in content are used to generate sentences with the same part of speech but different text content. .

[0112] The model training method provided in this application performs word segmentation on the cloned text and the original corpus to obtain two sets of word sequences. Under the premise of ensuring that the parts of speech are the same, the words in the two sequences are swapped to maximize the difference, thereby generating a new reconstructed text. This realizes the automated reconstruction of cloned text corresponding to unqualified samples, avoids manual intervention in the reconstruction process, and improves the efficiency of the reconstruction. The reconstructed text is then used to iteratively generate new audio samples for recognition and evaluation, which helps to gradually improve the quality of cloned audio and avoids low-quality cloned audio samples with semantic mismatch from entering the subsequent processing flow.

[0113] The model training method provided in this application can be executed by a model training device. This application uses an example of a model training device executing the model training method to illustrate the model training device provided in this application.

[0114] This application also provides a model training device.

[0115] Figure 6 This is a schematic diagram of the structure of a model training device provided in some embodiments of this application. For example... Figure 6 As shown, the model training device 600 includes: The generation module 601 is used to obtain cloned audio samples generated by the sound cloning model using the original corpus, wherein the sound cloning model is trained based on the reference text and the reference audio corresponding to the reference text. The filtering module 602 is used to identify the cloned text corresponding to the cloned audio sample and compare the cloned text with the original corpus to filter the cloned audio samples to obtain qualified samples and unqualified samples. Evaluation module 603 is used to evaluate the qualified samples based on the reference audio by using multiple evaluation models of ensemble learning, to obtain the evaluation results output independently by each evaluation model, and to filter the qualified samples by voting to obtain target samples to construct a training dataset. Training module 604 is used to perform distillation training on the sound synthesis model using the training dataset to obtain the trained sound synthesis model; The evaluation module 603 is further configured to evaluate the synthesized audio output by the trained sound synthesis model based on the reference audio through multiple evaluation models learned through ensemble learning. If the evaluation result of the synthesized audio does not meet the conditions, the process returns to the step of distilling the sound synthesis model using the training dataset and continues until the evaluation result of the synthesized audio meets the conditions.

[0116] In some embodiments, the evaluation module 603 is used to: The qualified samples are evaluated using the first evaluation model based on the reference audio, and the evaluation results are obtained. The qualified samples are evaluated using the second evaluation model to obtain the evaluation results; The qualified samples are evaluated using a third evaluation model based on their short-time energy (STE) and zero crossover rate (ZCR) characteristics, resulting in an evaluation outcome.

[0117] In some embodiments, the step of evaluating the qualified sample based on the reference audio using a first evaluation model to obtain an evaluation result includes: The MFCC feature values ​​of the qualified samples and the MFCC feature values ​​of the reference audio are obtained respectively. The MFCC feature values ​​of the qualified samples and the MFCC feature values ​​of the reference audio are reduced in dimensionality by using T-distributed random nearest neighbor embedding (T-SNE). Calculate the Mahalanobis distance between the MFCC eigenvalues ​​of the qualified samples after dimensionality reduction and the MFCC eigenvalues ​​of the reference audio. The evaluation result of the qualified sample is determined based on the Mahalanobis distance and the preset distance threshold.

[0118] In some embodiments, the step of evaluating the qualified samples using a second evaluation model to obtain evaluation results includes: The qualified samples are input into a pre-trained second evaluation model, which outputs the average opinion score (MOS) of the qualified samples; the second evaluation model includes a multi-layer Mamba network. The evaluation result of the qualified sample is determined based on the average opinion score and the first scoring threshold.

[0119] In some embodiments, the evaluation of the qualified samples using a third evaluation model, based on the short-time energy (STE) characteristic and zero crossover rate (ZCR) characteristic, yields evaluation results, including: The qualified sample is segmented into at least one audio frame according to the preset frame length, and the STE feature and ZCR feature of each audio frame are calculated. The noise score of the qualified sample is calculated based on the STE and ZCR features of each audio frame. The evaluation result of the qualified sample is determined based on the noise score and the second scoring threshold.

[0120] In some embodiments, the evaluation module 603 is used to: If there are no multiple evaluation results that are not "fail" in the independent output evaluation results, the qualified sample is determined as the target sample.

[0121] In some embodiments, the model training device 600 further includes a reconstruction module, which is used for If the cloned audio sample is unqualified, reconstructed text is generated based on the cloned text corresponding to the unqualified sample; Using the reconstructed text as the original corpus, the process returns to the step of obtaining cloned audio samples generated using the original corpus through the sound cloning model to continue execution.

[0122] In some embodiments, generating reconstructed text based on the text corresponding to the non-compliant samples includes: The cloned text and the original corpus are segmented into words respectively to obtain the first segmented word sequence corresponding to the cloned text and the second segmented word sequence corresponding to the original corpus. When the parts of speech of each corresponding word in the first word segmentation sequence and the second word segmentation sequence are the same, the words in the first word segmentation sequence and the second word segmentation sequence are swapped to maximize the difference between the first word segmentation sequence and the swapped first word segmentation sequence. The reconstructed text is generated based on the first word segmentation sequence after the exchange.

[0123] The model training device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0124] The model training device in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0125] The model training apparatus provided in this application embodiment can implement all the processes implemented in the above model training method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0126] Figure 7 These are schematic diagrams of the structure of an electronic device provided in some embodiments of this application. In some embodiments, such as Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 801, it implements the various processes of the above-described model training method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0127] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0128] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described model training method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0129] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0130] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described model training method.

[0131] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0132] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model training method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0133] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0134] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0136] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0137] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0138] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A model training method, characterized in that, include: Obtain cloned audio samples generated from the original corpus using a sound cloning model, wherein the sound cloning model is trained based on a reference text and the reference audio corresponding to the reference text. The cloned text corresponding to the cloned audio sample is identified, and the cloned text is compared with the original corpus to filter the cloned audio samples to obtain qualified and unqualified samples; Based on the reference audio, the qualified samples are evaluated by multiple evaluation models through ensemble learning, and the evaluation results output independently by each evaluation model are obtained. The qualified samples are then screened by voting to obtain target samples to construct a training dataset. The sound synthesis model is trained by distillation using the training dataset to obtain the trained sound synthesis model. Based on the reference audio, the synthesized audio output by the trained sound synthesis model is evaluated using multiple evaluation models learned through ensemble learning. If the evaluation result of the synthesized audio does not meet the conditions, the process returns to the step of distilling the sound synthesis model using the training dataset and continues until the evaluation result of the synthesized audio meets the conditions.

2. The method according to claim 1, characterized in that, The process of evaluating the qualified samples based on the reference audio using multiple evaluation models through ensemble learning, and obtaining the evaluation results independently output by each evaluation model, includes: The qualified samples are evaluated using the first evaluation model based on the reference audio, and the evaluation results are obtained. The qualified samples are evaluated using the second evaluation model to obtain the evaluation results; The qualified samples are evaluated using a third evaluation model based on their short-time energy (STE) and zero crossover rate (ZCR) characteristics, resulting in an evaluation outcome.

3. The method according to claim 1 or 2, characterized in that, The step of selecting target samples from qualified samples through voting to construct a training dataset includes: If there are no multiple evaluation results that are not "fail" in the independent output evaluation results, the qualified sample is determined as the target sample.

4. The method according to claim 2, characterized in that, The evaluation of the qualified samples based on the reference audio using the first evaluation model to obtain the evaluation result includes: The MFCC feature values ​​of the qualified samples and the MFCC feature values ​​of the reference audio are obtained respectively. The MFCC feature values ​​of the qualified samples and the MFCC feature values ​​of the reference audio are reduced in dimensionality by using T-distributed random nearest neighbor embedding (T-SNE). Calculate the Mahalanobis distance between the MFCC eigenvalues ​​of the qualified samples after dimensionality reduction and the MFCC eigenvalues ​​of the reference audio. The evaluation result of the qualified sample is determined based on the Mahalanobis distance and the preset distance threshold.

5. The method according to claim 2, characterized in that, The evaluation of the qualified samples using the second evaluation model to obtain evaluation results includes: The qualified samples are input into the pre-trained second evaluation model, which outputs the average opinion score (MOS) of the qualified samples; the second evaluation model includes a multi-layer Mamba network. The evaluation result of the qualified sample is determined based on the average opinion score and the first scoring threshold.

6. The method according to claim 2, characterized in that, The qualified samples are evaluated using a third evaluation model based on their short-time energy (STE) and zero-crossing rate (ZCR) characteristics to obtain evaluation results, including: The qualified sample is segmented into at least one audio frame according to the preset frame length, and the STE feature and ZCR feature of each audio frame are calculated. The noise score of the qualified sample is calculated based on the STE and ZCR features of each audio frame. The evaluation result of the qualified sample is determined based on the noise score and the second scoring threshold.

7. The method according to claim 1, characterized in that, The method further includes: If the cloned audio sample is unqualified, reconstructed text is generated based on the cloned text corresponding to the unqualified sample; Using the reconstructed text as the original corpus, the process returns to the step of obtaining cloned audio samples generated using the original corpus through the sound cloning model to continue execution.

8. The method according to claim 7, characterized in that, The process of generating reconstructed text based on the cloned text corresponding to the non-compliant samples includes: The cloned text and the original corpus are segmented into words respectively to obtain the first segmented word sequence corresponding to the cloned text and the second segmented word sequence corresponding to the original corpus. When the parts of speech of each corresponding word in the first word segmentation sequence and the second word segmentation sequence are the same, the words in the first word segmentation sequence and the second word segmentation sequence are swapped to maximize the difference between the first word segmentation sequence and the swapped first word segmentation sequence. The reconstructed text is generated based on the first word segmentation sequence after the exchange.

9. A model training device, characterized in that, include: A generation module is used to obtain cloned audio samples generated by a sound cloning model using the original corpus. The sound cloning model is trained based on a reference text and the reference audio corresponding to the reference text. The filtering module is used to identify the cloned text corresponding to the cloned audio sample and compare the cloned text with the original corpus to filter the cloned audio samples to obtain qualified samples and unqualified samples. The evaluation module is used to evaluate the qualified samples based on the reference audio by using multiple evaluation models learned through ensemble learning, obtain the evaluation results output independently by each evaluation model, and filter the qualified samples by voting to obtain target samples to construct a training dataset. The training module is used to perform distillation training on the sound synthesis model using the training dataset to obtain the trained sound synthesis model. The evaluation module is further configured to evaluate the synthesized audio output by the trained sound synthesis model based on the reference audio and through multiple evaluation models learned through ensemble learning. If the evaluation result of the synthesized audio does not meet the conditions, the module returns to the step of distilling the sound synthesis model using the training dataset and continues execution until the evaluation result of the synthesized audio meets the conditions.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the model training method as described in any one of claims 1-8.

Images