Speech synthesis adaptation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a speech synthesis adaptation method and device, equipment and a storage medium, which can be applied to customer service systems in the insurance fields of finance, medical treatment and the like. According to the method, basic training data and fine tuning data can be obtained firstly. And converting the training text into international phonetic symbols, and learning a cross-language general mapping relation on a phoneme level to obtain general mapping data. The base model is then trained based on the generic mapping data, and fine-tuned using the fine-tuning data. And performing online preference optimization on the basic model according to the prompt set and the multi-target reward function to obtain an adaptive model. According to the method, model fine tuning can be performed through multi-language IPA basic model training in combination with a small amount of paired data in a target language environment, and GRPO online preference optimization based on multi-index rewards is performed, so that the intelligibility, speaker consistency and tone quality of low-resource language synthesis are improved on the premise that large-scale parallel corpora are not needed.

Description

Technical Field

[0001] This application relates to the field of insurance service system technology, and in particular to a speech synthesis adaptation method, apparatus, device and storage medium. Background Technology

[0002] Text-to-speech (TTS) technology is a data conversion technology that transforms text content into speech signals for output. It can be applied to interactive scenarios such as voice broadcasting in the financial and healthcare insurance sectors, intelligent customer service systems, and intelligent voice assistants. TTS technology performs grammatical, semantic, and prosodic analysis on the text to be converted, uses acoustic models to match speech units, and then concatenates the speech units into an audio format speech signal for playback or storage.

[0003] For different user groups, TTS technology can be used to convert text data into speech signals in different language formats. For example, TTS applications developed for English-speaking environments can convert English text data into English speech signals. Furthermore, thanks to the advantages of large-scale paired corpora and targeted large-scale model development processes in English-speaking environments, TTS technology can achieve good naturalness, intelligibility, and prosodic consistency in speech output in English and other language environments.

[0004] However, for applications using less commonly spoken languages ​​or dialects with limited resources, the scarcity and high cost of high-quality paired speech-text corpora, along with factors such as linguistic differences and speaker diversity, result in weak cross-language generalization capabilities. While cross-language transfer methods that directly fine-tune multilingual models can improve the generalization ability of TTS technology, they still have significant shortcomings in intelligibility, speaker consistency, and sound quality. Therefore, TTS technology still requires large-scale parallel corpora in applications using less commonly spoken languages ​​or dialects. Summary of the Invention

[0005] In view of this, embodiments of this application provide a speech synthesis adaptation method, apparatus, device, and storage medium to solve the problem that the speech synthesis process requires large-scale parallel corpora.

[0006] According to a first aspect of this application, a speech synthesis adaptation method is provided, the method comprising: Acquire basic training data and fine-tuning data. The basic training data includes training texts in multiple languages; the fine-tuning data is paired data for the target language environment. Generate general mapping data based on the basic training data. The general mapping data is obtained by converting the training text into the International Phonetic Alphabet and learning cross-language general mapping relationships at the phoneme level. The basic model is trained based on the general mapping data. The basic model is an automatic speech recognition model that adopts a non-autoregressive text encoder and an autoregressive decoder architecture. Use the fine-tuning data to fine-tune the model parameters of the base model; Based on the cue set and the multi-objective reward function, online preference optimization is performed on the fine-tuned base model to obtain an adapted model; the cue set includes target language text and reference audio; the multi-objective reward function is constructed based on automatic speech recognition, speaker verification, and perceptual evaluation of speech quality.

[0007] According to a second aspect of this application, a speech synthesis adaptation device is provided, the device comprising: The data acquisition module is used to acquire basic training data and fine-tuning data. The basic training data includes training texts in multiple languages; the fine-tuning data is paired data for the target language environment. The data mapping module is used to generate general mapping data based on the basic training data. The general mapping data is obtained by converting the training text into the International Phonetic Alphabet and learning cross-language general mapping relationships at the phoneme level. The basic model training module is used to train a basic model based on the general mapping data. The basic model is an automatic speech recognition model that adopts a non-autoregressive text encoder and an autoregressive decoder architecture. The model fine-tuning module is used to fine-tune the model parameters of the base model using the fine-tuning data; An online preference optimization module is used to perform online preference optimization on the fine-tuned base model based on a cue set and a multi-objective reward function to obtain an adapted model; the cue set includes target language text and reference audio; the multi-objective reward function is constructed based on automatic speech recognition, speaker verification, and perceptual evaluation of speech quality.

[0008] According to a third aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described speech synthesis adaptation method.

[0009] According to a fourth aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described speech synthesis adaptation method.

[0010] By employing the above technical solutions, embodiments of this application provide a speech synthesis adaptation method, apparatus, device, and storage medium. The method first acquires basic training data and fine-tuning data. Then, it converts the training text into International Phonetic Alphabet (IPA) symbols and learns cross-language universal mapping relationships at the phoneme level to obtain universal mapping data. Next, it trains a basic model based on the universal mapping data and fine-tunes the basic model using the fine-tuning data. Finally, it performs online preference optimization on the basic model based on a prompt set and a multi-objective reward function to obtain an adapted model. This method can adapt TTS models for low-resource language environments. It trains a multilingual IPA basic model, fine-tunes the model using paired data from a small target language environment, and then performs online preference optimization based on a multi-index reward GRPO. This improves the intelligibility, speaker consistency, and sound quality of low-resource language synthesis without requiring a large-scale parallel corpus.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic flowchart of the speech synthesis adaptation method provided in the embodiments of this application; Figure 2 A schematic diagram of the basic model architecture provided for embodiments of this application; Figure 3 A schematic diagram of the GRPO online preference optimization architecture provided in the embodiments of this application; Figure 4 This is a schematic diagram of the output adaptation model process provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the process of outputting a speech signal using an adaptation model, as provided in an embodiment of this application. Figure 6 This is a schematic diagram of the speech synthesis adapter provided in an embodiment of this application. Detailed Implementation

[0013] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0014] In this embodiment, the speech synthesis adapter can be used to construct a speech synthesis model suitable for a specific language environment. The speech synthesis model is an artificial intelligence model that uses Text-to-Speech (TTS) technology to convert text content into speech signals for information output. The speech synthesis model can be applied to interactive scenarios such as voice broadcasting in the financial and medical insurance sectors, intelligent customer service systems, and intelligent voice assistants.

[0015] Speech synthesis models can be based on TTS technology to perform grammatical, semantic and prosodic analysis on the text to be converted, and combine it with an acoustic model to match phoneme units. The matched phoneme units are then concatenated into an audio format speech signal for speech playback or storage.

[0016] For example, in a customer service system for financial insurance, an intelligent question-answering robot based on a speech synthesis model can be set up. After a user inputs a text or voice inquiry through the intelligent question-answering robot, the robot can call a large language model to perform knowledge retrieval, and simultaneously call a speech synthesis model to generate a response voice signal based on the knowledge retrieval results.

[0017] Since speech synthesis models can support users to interact using natural language, and different users use different languages ​​and input different types of speech, speech synthesis models can be applied to different user groups to convert text data into speech signals in different language forms.

[0018] For example, a speech synthesis model developed for an English-speaking environment can convert English text data into English speech signals. Furthermore, thanks to the advantages of a large-scale paired corpus and a targeted model development process in English-speaking environments, the speech synthesis model can achieve good naturalness, intelligibility, and prosodic consistency in speech output in English and other language environments.

[0019] However, for application scenarios involving less common languages ​​or dialects with limited resources, the cross-language generalization ability is weak due to the scarcity and high cost of high-quality paired speech-text corpora, as well as the influence of factors such as linguistic differences and speaker diversity.

[0020] Therefore, in some embodiments, a speech synthesis model suitable for the target language environment can be generated by directly fine-tuning a multilingual model. This involves selecting a pre-trained TTS model that supports multiple languages ​​as the base model. Then, speech data from the target language environment and corresponding text-to-speech transcriptions are collected to obtain training data for fine-tuning. The collected fine-tuning training data is then used to fine-tune the base model, and optimization strategies such as transfer learning and knowledge distillation are combined to improve the model's performance, resulting in a speech synthesis model suitable for the target language environment.

[0021] It is evident that while speech synthesis models obtained through cross-language transfer can improve the generalization ability of TTS technology, they still have significant shortcomings in intelligibility, speaker consistency, and sound quality. This means that speech synthesis models obtained through fine-tuning still require large-scale parallel corpora in application scenarios involving minority languages ​​or dialects.

[0022] In some embodiments, Automatic Speech Recognition (ASR) technology can be used to pre-train a large multilingual model in a low-resource language environment, thereby obtaining a more reliable ASR model. For example, speech data can be collected in a low-resource language environment and converted into text data using ASR technology. Then, the speech data and text data can be used as paired data to pre-train a large multilingual model such as Whisper or CommonVoice, thereby obtaining a speech synthesis model.

[0023] However, because the generated speech synthesis model lacks an online feedback loop, it cannot dynamically optimize the generated results. Furthermore, the speech synthesis model evaluation process only considers a single evaluation metric, making it difficult to improve the intelligibility and speaker consistency of low-resource languages.

[0024] To address the issue of requiring large-scale parallel corpora in speech synthesis, some embodiments of this application provide a speech synthesis adaptation method. This method can perform multi-stage adaptation for TTS processes in low-resource language environments. By training a multilingual IPA base model, fine-tuning with a small amount of paired data, and then using an online preference optimization method based on multi-index reward group relative policy optimization (GRPO), it can significantly improve the intelligibility, speaker consistency, and sound quality of low-resource language synthesis without requiring large-scale parallel corpora. Furthermore, it outperforms offline alignment methods such as direct preference optimization (DPO) in high-resource language environments.

[0025] The speech synthesis and adaptation method described herein can be applied to electronic devices with data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. For ease of description, this embodiment uses an electronic device as the execution subject of the method. It should be understood that the method can also be applied to other types of execution subjects, which are not illustrated in this embodiment. Figure 1 As shown, the method includes: S101. Obtain basic training data and fine-tuning data.

[0026] When performing speech synthesis adaptation, basic training data can be acquired first, which includes training text in multiple languages. Furthermore, the training text can be labeled with speech tags to form paired data of text and speech in multiple languages.

[0027] Basic training data can be obtained from databases with high-resource language environments. For example, to obtain basic training data, an electronic device can generate a basic training data retrieval request and send it to multilingual corpora such as Common Voice or LJSpeech. After receiving the basic training data, the multilingual corpus can feed it back to the electronic device.

[0028] During speech synthesis adaptation, fine-tuning data can also be acquired, which consists of small-scale paired data of the target language environment. This fine-tuning data can be obtained from databases of low-resource language environments. For example, when speech synthesis is adapted to a language A environment of a less commonly spoken language, small-scale paired data of text corpora ranging from 30 minutes to 5 hours of sample speech can be collected from language A environment as fine-tuning data.

[0029] In some embodiments, text and speech data obtained from a specific language environment can also be preprocessed to generate training data in a format that conforms to the model input. For example, when acquiring basic training data and fine-tuning data, after collecting text and corresponding phoneme annotation data in multiple languages, the collected data can be preprocessed, and then basic training data can be generated based on the preprocessed data.

[0030] In the preprocessing stage, text cleaning can be performed first to remove irrelevant characters such as redundant spaces and punctuation marks. Then, text normalization converts numbers, dates, abbreviations, etc., into a data format readable by the model. Next, phoneme annotation is performed, using tools like espeak and g2p to convert the text into a phoneme sequence and then labeling it. Finally, alignment is used to align the text and phoneme sequence, ensuring each character or phoneme has a corresponding label.

[0031] Similarly, for low-resource language environments, data can be collected in the same way, and the collected data can be preprocessed to generate fine-tuned data. The only difference is the specific language type of the collected data and the data scale, which is different from the basic training data. This will not be elaborated here.

[0032] S102. Generate general mapping data based on the basic training data.

[0033] After acquiring the basic training data, general mapping data can be generated based on it. This general mapping data is obtained by converting the training text into International Phonetic Alphabet (IPA) symbols and learning cross-language general mapping relationships at the phoneme level.

[0034] In some embodiments, when generating general mapping data based on basic training data, training text can be extracted from the basic training data, and a set of phonetic symbols can be obtained by converting the training text into International Phonetic Alphabet (IPA). Then, a byte-level word segmenter is used to segment the training text to generate a word segmentation set. This word segmentation set includes multiple keywords represented by phonetic symbols. Finally, based on the general mapping relationship between phonetic symbols and phoneme units in the phonetic symbol set, associated phoneme units are set for the keywords to generate general mapping data.

[0035] For example, training text in the basic training data can be uniformly converted into International Phonetic Alphabet (IPA) symbols and modeled using a byte-level segmenter with 256 symbols, thereby learning a universal cross-language mapping at the phoneme level. To do this, electronic devices can call IPA conversion tools, such as epitran and g2p, and use these tools to convert the text into a set of IPA symbols. Since the basic training data can contain data from multiple languages, it is necessary to process the text in each language separately. Language detection tools such as langdetect can be used to identify the language type of the training text, and then an IPA conversion tool appropriate for that language type can be used for conversion.

[0036] After converting the training text to the International Phonetic Alphabet (IPA), byte-level segmenters such as Byte Pair Encoding and BPE can be used to convert the training text into byte-level symbols to generate a segmented word set. Then, based on the universal mapping relationship between phonetic symbols and phoneme units in the phonetic symbol set, associated phoneme units are set for keywords to generate universal mapping data. If a vocabulary of 256 symbols is constructed using the trained segmenter, the segmented data can be saved as universal mapping data and used as sample data in the model training process for subsequent use.

[0037] S103. Train the basic model based on general mapping data.

[0038] After generating universal mapping data, a basic model can be trained based on the universal mapping data. That is, electronic devices can use the universal mapping data to train models to obtain a basic model suitable for multilingual environments. The basic model is an automatic speech recognition model that adopts a non-autoregressive text encoder (NAR encoder) and an autoregressive decoder (AR decoder) architecture.

[0039] like Figure 2 As shown, the non-autoregressive text encoder (NAR encoder) generates its output by encoding each part of the encoder simultaneously and in parallel, without depending on previously generated elements. The NAR encoder can transform the input sequence into a representation containing rich information, namely a context representation vector.

[0040] When generating the output sequence, the autoregressive decoder (AR decoder) can generate each element sequentially. Each new element depends on all previously generated elements. The autoregressive decoder can utilize the context representation vector generated by the encoder to generate the target sequence.

[0041] The base model, which is a multilingual base TTS model based on an architecture of non-autoregressive text encoder and autoregressive decoder, can be optimized by using the next frame prediction loss of the non-autoregressive text encoder and autoregressive decoder to obtain a base model that is independent of language type.

[0042] In some embodiments, when training a base model based on general mapping data, the text content in the general mapping data can be input into the trained base model, and the input text can be encoded into a context representation vector by a non-autoregressive text encoder in the base model. Then, an autoregressive decoder in the base model is used to generate a phoneme sequence based on the context representation vector. The phoneme sequence includes the generated phonemes and the predicted phonemes for the current stage.

[0043] For example, during the training of the base model, the parameters of the non-autoregressive text encoder and autoregressive decoder in the trained model can be initialized first. Then, through forward propagation, the input text is passed through the non-autoregressive text encoder to obtain the context representation vector. Finally, the context representation vector and the generated phoneme sequence are input into the autoregressive decoder to predict the next phoneme.

[0044] The phoneme distribution is then predicted based on the phoneme sequence, where the phoneme distribution includes multiple phoneme positions. For each phoneme position, the cross-entropy loss between the phoneme distribution and the phoneme units in the general mapping data can be calculated, and the base model is iteratively trained based on the cross-entropy loss.

[0045] In some embodiments, to enable iterative training of the base model, cross-entropy losses corresponding to multiple phoneme positions can be accumulated during training to obtain a base training loss. A base loss threshold is then obtained and compared with the base training loss. If the base training loss is greater than the threshold, it indicates that the current model has not yet converged. Therefore, the model parameters of the base model can be updated by backpropagating the base training loss. Phoneme distribution prediction is then performed based on the updated base model for iterative training.

[0046] Through multiple iterations of training, when the basic training loss is less than or equal to the basic loss threshold, or when the preset number of iterations is reached, it can be determined that the current basic model has been trained to convergence. Therefore, the current model parameters can be output to obtain the trained basic model.

[0047] For example, for each phoneme position t, the cross-entropy loss between the predicted phoneme distribution and the actual phonemes can be calculated. Then, by summing the losses for all positions, the basic training loss is obtained, serving as the total loss for the basic training phase. The model parameters are then updated via backpropagation, with the iterative objective of minimizing the total loss, using optimizers such as Adam to update the parameters of the basic model. Through training loops, forward propagation, loss calculation, and backpropagation are repeated until the model converges.

[0048] S104. Use fine-tuning data to fine-tune the model parameters of the base model.

[0049] After obtaining the base model through training, electronic devices can also use fine-tuning data to fine-tune the model parameters of the base model. That is, by inputting small-scale paired data into the base model and obtaining the output results of the base model, the model parameters of the base model are adjusted according to the output results.

[0050] For example, the base model can be fine-tuned using small-scale paired data ranging from 30 minutes to 5 hours on the target language, allowing the model to learn the phonological and prosodic features of the target language. During fine-tuning, low-resource data, i.e., fine-tuning data, can be upsampled and mixed with the base training data for mixed training to prevent catastrophic forgetting.

[0051] Therefore, in some embodiments, in order to fine-tune the base model, when fine-tuning the model parameters of the base model using fine-tuning data, mixed sample data can be generated by mixing the fine-tuning data with the base training data, and upsampling can be performed on the fine-tuning data in the mixed sample data to obtain resampled data. The proportion of fine-tuning data in the resampled data is greater than or equal to a preset proportion threshold.

[0052] The resampled data is then input into the base model to obtain its output. The output is then compared with the speech labels in the fine-tuned data to calculate the fine-tuning training loss. The model parameters of the base model are then adjusted based on this fine-tuning training loss. The fine-tuning training loss is the training loss calculated based on the resampled data of the next frame.

[0053] For example, when fine-tuning the base model, resampled data can be obtained through data mixing and upsampling. Specifically, during data mixing, small-scale paired data in the target language can be combined with multilingual data to form a hybrid dataset. Because the hybrid dataset contains data from multiple languages, using it to fine-tune the base model allows it to maintain its generalization ability in multilingual environments.

[0054] When upsampling low-resource data, the Synthetic Minority Over-sampling Technique (SMOTE) can be used to upsample small-scale paired data of the target language, so that it occupies a certain proportion in the mixed dataset.

[0055] By fine-tuning the base model on a mixed dataset, the goal can be set to better adapt the base model to the phonological and prosodic features of the target language. The base model is then optimized using the next-frame prediction loss to ensure accurate prediction of the next phoneme.

[0056] During fine-tuning of the base model, the optimizer and loss function can be initialized. The text in the mixed dataset is then passed through a non-autoregressive text encoder to obtain a context representation. For each phoneme position t, the cross-entropy loss between the predicted phoneme distribution and the true phonemes is calculated. Then, through backpropagation, the model parameters are updated using optimizers such as Adam to minimize the total loss. Therefore, through a training loop, forward propagation, loss calculation, and backpropagation are repeated until the model converges.

[0057] To prevent catastrophic forgetting during fine-tuning of the base model, each training batch should include fine-tuning data for the target language and base training data for multiple languages. This prevents overfitting to the target language while maintaining generalization ability to other languages. Furthermore, during model fine-tuning, techniques such as Dropout and L2 regularization can be used to prevent overfitting, and KL divergence constraints can be applied to ensure that the output distribution of the fine-tuned model does not differ significantly from the base model.

[0058] During the fine-tuning of the base model, metrics such as BLEU score and WER (Word Error Rate) can be established to evaluate the model's generation quality. For the target language environment, phonological and prosodic feature evaluation metrics can be used, such as Character Error Rate (CER), Structural Similarity Index Measure (SSIM), and Perceptual Evaluation of Speech Quality (PESQ). This allows the fine-tuned base model to serve as a reference policy for subsequent reinforcement learning.

[0059] S105. Based on the prompt set and the multi-objective reward function, perform online preference optimization on the fine-tuned base model to obtain a suitable model.

[0060] The fine-tuned base model can then undergo online preference optimization. This involves performing online preference optimization on the fine-tuned base model based on a cue set and a multi-objective reward function to obtain a suitable model. The cue set includes target language text and reference audio; the multi-objective reward function is constructed based on automatic speech recognition, speaker verification, and perceptual evaluation of speech quality.

[0061] In some embodiments, when performing online preference optimization on the fine-tuned base model, prompt construction can be performed first. This involves acquiring the target language text and speaker reference audio, and constructing a prompt set based on the target language text and reference audio. The prompt set includes optimized prompts for non-parallel speech and text resources.

[0062] For example, when constructing prompts, a set of prompts can be built, consisting of target language text and speaker reference audio. The audio content of the reference audio may not match the target language text, thus allowing for the utilization of non-parallel speech and text resources for group-based strategy optimization. Specifically, based on a temperature 0.7 polynomial sampling method, each mini-batch contains 64 prompts, and each prompt can generate 12 audio samples.

[0063] A multi-objective reward function is then established. This multi-objective reward function represents a weighted sum of multiple reward metrics. These multiple reward metrics include at least an Automatic Speech Recognition (ASR) reward metric, a Speaker Verification (SV) reward metric, and a perceptual assessment of speech quality reward metric.

[0064] For example, such as Figure 3As shown, in online preference optimization based on Group Relative Policy Optimization (GRPO), three reward metrics can be calculated for each generated sample, and the calculated reward metrics can be weighted and combined to form the total reward. The total reward can then be expressed as: R=w cer ×R cer +w ssim ×R ssim +w pesq ×R pesq ; Among them, w cer w ssim And the weights for character error rate (CER), structural similarity index (SSIM), and perceptual evaluation of speech quality (PESQ), respectively, such as w cer =w ssim =0.45, w pesq =0.1. R cer The character error rate reward score can be obtained by calculating the character error rate based on Whisper Large V3 and linearly normalizing it to the [0,1] interval. R ssim The reward score representing the structural similarity index can be obtained by calculating and normalizing the cosine similarity between the contextual audio and the synthesized audio embeddings based on Titanet-large. R pesq The reward score representing the perceived quality of speech can be obtained by dividing by a no-reference neural PESQ estimator and normalizing by 4.5.

[0065] After constructing a multi-objective reward function, the base model can be optimized based on the multi-objective reward function to generate a suitable model. In some embodiments, to generate a suitable model, when optimizing the base model based on the multi-objective reward function, the sample rewards of the cue set can be calculated according to the multi-objective reward function, and then a group baseline can be set based on the sample rewards, wherein the group baseline is the average of multiple sample rewards for the same optimization cue. Then, the dominance value is calculated based on the group baseline, a preset probability model is obtained, and an optimization loss is calculated based on the preset probability model and the dominance value, thereby iteratively training the base model based on the optimization loss to generate a suitable model.

[0066] For example, after constructing the cue and establishing the multi-objective reward function, the advantage value can be calculated and the objective function set based on the cue set and the multi-objective reward function. Then, the average reward of K samples for the same cue can be calculated. µ i As the group baseline, and then according to A i,k = r i,k - µ i Calculate the advantage value. Therefore, the optimization objective can be set as:

[0067] in, L GRPO This represents the loss function value during the optimization phase; M This represents the total number of samples; K Indicates the number of sample rewards; A i,k The dominance value, also known as the weighting factor, represents the first... i The first sample k The relative importance or advantage of each output; π θ ( y i,k | x i ) is a probabilistic model with parameters. θ , indicating that given input x i Under the conditions, the first i The sample generates the first k Output y i,k The probability of; log It represents the natural logarithm.

[0068] Based on the set optimization objective, during group relative strategy optimization, the output results of K samples can be collected for each input (prompt), and a reward value can be calculated for each sample based on the reward function, thus achieving evaluation based on the output quality of the samples and relevant standards. Then, the advantage value is calculated, that is, for each input, the average reward of its K samples is calculated. µ i And by assigning a reward value to each sample. r i,k Subtract the average reward of the input µ i To calculate the advantage value A i,k .

[0069] During the model prediction process, a fine-tuned base model can be used. π θ Output for each sample y i,k Given input x i The probability is predicted. Then the loss value in the optimization stage is calculated, that is, for each sample, the log probability predicted by the model is calculated. logπ θ ( y i,k | x iThen, the log probability of each sample is multiplied by its dominance value to obtain the weighted log probability.

[0070] The average weighted log probability is obtained by summing the weighted log probabilities of all samples and dividing the sum by the total number of samples, MK. The model parameters are then optimized by minimizing this average weighted log probability. Optimization algorithms such as gradient descent are used to update the model parameters based on the calculated loss. This process is repeated iteratively until the model converges or reaches a preset number of iterations, resulting in a suitable model.

[0071] By applying the technical solutions of the above embodiments, the speech synthesis adaptation method described in the above embodiments can automatically provide preference signals and optimize the TTS model online using ASR, SV, sound quality estimator and other mechanisms in scenarios with very little paired data, non-parallel speech text and multi-index optimization, so that the adaptation model can improve the intelligibility and speaker consistency in low-resource language environments.

[0072] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a speech synthesis adaptation method, such as... Figure 4 As shown, the method includes: S201. Obtain training and validation data; S202. Input the training and validation data into the adaptation model to obtain the validation speech signal output by the adaptation data. S203. By comparing and verifying the speech signal and speech label, generate model evaluation indicators; S204. If the model evaluation index meets the preset evaluation conditions, output the model parameters that adapt to the model.

[0073] Before outputting the adapted model, a model evaluation can be performed. For this purpose, the electronic device can acquire training and validation data, which includes validation text in the target language environment and speech tags labeled for the validation text.

[0074] The training and validation data are then input into the adaptation model to obtain the validation speech signal output by the adaptation data. By comparing the validation speech signal with the speech labels, model evaluation metrics are generated. These metrics include at least one of the following: character error rate, structural similarity index, perceptual speech quality assessment parameters, speech quality index, and average opinion score. If the model evaluation metrics meet preset evaluation conditions, the model parameters of the adaptation model are output.

[0075] For example, training and validation data can be used to infer and evaluate the adapted model. The adapted model after GRPO can then generate high-quality speech under low-resource language prompts. Evaluation metrics such as CER, SSIM, PESQ, and Squim-MOS can be used during the evaluation process. Clearly, for CER, lower is better; while for SSIM, higher is better.

[0076] By applying the technical solutions of the above embodiments, the speech synthesis adaptation method described in the above embodiments can, after obtaining the adaptation model, input training and validation data into the adaptation model to obtain the validation speech signal output by the adaptation data, and then compare the validation speech signal with the speech label to generate multivariate model evaluation metrics. Only when the model evaluation metrics meet the preset evaluation conditions are the model parameters of the adaptation model output, so that the output adaptation model can generate high-quality speech under low-resource language prompts.

[0077] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a speech synthesis adaptation method, such as... Figure 5 As shown, the method includes: S301. Obtain the text data to be synthesized; S302. Input the text data to be synthesized into the adaptation model; S303. Using a non-autoregressive text encoder with an adaptive model, the synthetic text data is encoded into a context representation vector; S304. Using an autoregressive decoder with an adaptive model, phoneme sequences are generated based on context representation vectors. S305, based on phoneme sequence matching phoneme units, and generating output speech data by assembling phoneme units.

[0078] After outputting the model parameters of the adaptation model, the electronic device can deploy the adaptation model on any data processing carrier. The deployed adaptation model is then used for speech synthesis. This involves first acquiring the text data to be synthesized and inputting it into the adaptation model. The adaptation model then uses a non-autoregressive text encoder to encode the synthesized text data into a context representation vector, and an autoregressive decoder to generate a phoneme sequence based on the context representation vector. Finally, it matches phoneme units based on the phoneme sequence and generates output speech data by assembling the phoneme units.

[0079] By applying the technical solutions of the above embodiments, the speech synthesis adaptation method described in the above embodiments can obtain output speech data in the target language environment by inputting the text data to be synthesized into the adaptation model. Since the adaptation model is trained using a multilingual IPA basic model, fine-tuned using a small amount of paired data in the target language environment, and optimized using GRPO online preference optimization based on multi-index rewards, it can obtain output speech data with high intelligibility, speaker consistency, and sound quality in the target language environment, thus improving the user experience.

[0080] In some embodiments, as a specific implementation of the speech synthesis adaptation method in the above embodiments, some embodiments of this application also provide a speech synthesis adaptation device, such as... Figure 6 As shown, the device includes: The data acquisition module is used to acquire basic training data and fine-tuning data. The basic training data includes training text in multiple languages; the fine-tuning data is paired data for the target language environment. The data mapping module is used to generate general mapping data based on the basic training data. The general mapping data is obtained by converting the training text into the International Phonetic Alphabet and learning cross-language general mapping relationships at the phoneme level. The basic model training module is used to train a basic model based on general mapping data. The basic model is an automatic speech recognition model that adopts a non-autoregressive text encoder and an autoregressive decoder architecture. The model fine-tuning module is used to fine-tune the model parameters of the base model using fine-tuning data; The online preference optimization module is used to perform online preference optimization on the fine-tuned base model based on the cue set and the multi-objective reward function to obtain an adapted model; the cue set includes target language text and reference audio; the multi-objective reward function is constructed based on automatic speech recognition, speaker verification, and perceptual evaluation of speech quality.

[0081] By applying the technical solutions of the above embodiments, the speech synthesis adaptation device described in the above embodiments can, after the data acquisition module acquires basic training data and fine-tuning data, convert the training text into International Phonetic Alphabet (IPA) through the data mapping module, and learn cross-language universal mapping relationships at the phoneme level to obtain universal mapping data. Then, the basic model training module trains a basic model based on the universal mapping data, and the model fine-tuning module uses fine-tuning data to fine-tune the basic model. The online preference optimization module performs online preference optimization on the basic model based on the prompt set and multi-objective reward function to obtain an adapted model. The device can perform TTS model adaptation for low-resource language environments. Through multilingual IPA basic model training, combined with a small amount of matching data in the target language environment for model fine-tuning, and then based on GRPO online preference optimization with multi-index rewards, it can improve the intelligibility, speaker consistency, and sound quality of low-resource language synthesis without the need for large-scale parallel corpora.

[0082] It should be noted that other corresponding descriptions of the functional units involved in the speech synthesis adaptation device provided in the embodiments of this application can be found in the corresponding descriptions in the speech synthesis adaptation method provided in the above embodiments, and will not be repeated here.

[0083] This application also provides a computer device, specifically a personal computer, server, network device, etc. The computer device includes a bus, processor, memory, and communication interface, and may also include input / output interfaces and a display device. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores location information. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0084] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0085] In one embodiment, a computer-readable storage medium is also provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0086] In one embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the methods described above.

[0089] Any references to memory, database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc.

[0090] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0091] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A speech synthesis adaptation method, characterized by, The method comprises: obtaining basic training data and fine-tuning data, the basic training data comprising training texts in multiple languages; and the fine-tuning data being paired data in a target language environment; generating general mapping data from the basic training data, the general mapping data being obtained by converting the training texts into international phonetic alphabets and learning cross-language general mapping relationships at a phoneme level; training a basic model based on the general mapping data, the basic model being an automatic speech recognition model adopting a non-autoregressive text encoder and an autoregressive decoder architecture; fine-tuning model parameters of the basic model using the fine-tuning data; performing online preference optimization on the fine-tuned basic model according to a prompt set and a multi-objective reward function to obtain an adapted model, the prompt set comprising target language texts and reference audio, and the multi-objective reward function being constructed based on automatic speech recognition, speaker verification and perceptual evaluation speech quality.

2. The method of claim 1, wherein, The method comprises: extracting the training texts from the basic training data; obtaining a set of phonetic alphabet symbols by converting the training texts into international phonetic alphabets; performing word segmentation on the training texts using a byte-level word segmenter to generate a segmented word set comprising a plurality of keywords represented by phonetic alphabet symbols; setting associated phoneme units for the keywords according to a general mapping relationship between phonetic alphabet symbols and phoneme units in the set of phonetic alphabet symbols to generate the general mapping data.

3. The method of claim 1, wherein, The method comprises: inputting text content in the general mapping data into the trained basic model; encoding the input text into context representation vectors by a non-autoregressive text encoder in the basic model; generating a phoneme sequence based on the context representation vectors using an autoregressive decoder in the basic model, the phoneme sequence comprising generated phonemes and predicted phonemes; predicting a phoneme distribution from the phoneme sequence, the phoneme distribution comprising a plurality of phoneme positions; calculating cross-entropy loss between the phoneme distribution and phoneme units in the general mapping data for each phoneme position; iteratively training the basic model based on the cross-entropy loss.

4. The method of claim 3, wherein, The method comprises: accumulating the cross-entropy loss corresponding to a plurality of phoneme positions to obtain a basic training loss; obtaining a basic loss threshold; updating model parameters of the basic model by back-propagating the basic training loss if the basic training loss is greater than the basic loss threshold; outputting current model parameters to obtain a trained basic model if the basic training loss is less than or equal to the basic loss threshold.

5. The method of claim 1, wherein, The method comprises: generating mixed sample data by mixing the fine-tuning data and the basic training data; performing up-sampling on the fine-tuning data in the mixed sample data to obtain resampled data, the proportion of the fine-tuning data in the resampled data being greater than or equal to a preset proportion threshold; input the resampling data into the base model, and obtain a fine-tuning training loss output by the base model; the fine-tuning training loss is a training loss calculated based on next-frame resampling data; adjust model parameters of the base model based on the fine-tuning training loss.

6. The method of claim 1, wherein, According to the prompt set and the multi-objective reward function, the fine-tuned base model is subjected to online preference optimization to obtain an adapted model, including: obtaining target language text and speaker reference audio; constructing a prompt set according to the target language text and the reference audio; the prompt set includes optimized prompts of non-parallel speech and text resources; setting a multi-objective reward function, which is used to represent a weighted sum of multiple reward indicators; at least one of the multiple reward indicators includes an automatic speech recognition reward indicator, a speaker verification reward indicator, and a perceptual evaluation speech quality reward indicator; optimizing the base model according to the multi-objective reward function to generate the adapted model.

7. The method of claim 6, wherein, According to the multi-objective reward function, the base model is optimized to generate the adapted model, including: calculating sample rewards of the prompt set according to the multi-objective reward function; setting a group baseline based on the sample rewards, the group baseline being an average value of multiple sample rewards of the same optimized prompt; calculating an advantage value according to the group baseline; obtaining a preset probability model, and calculating an optimization loss according to the preset probability model and the advantage value; iteratively training the base model according to the optimization loss to generate the adapted model.

8. A speech synthesis adaptation apparatus characterized by comprising: The device includes: a data acquisition module configured to acquire base training data and fine-tuning data, the base training data including training text in multiple languages; and the fine-tuning data being paired data in a target language environment; a data mapping module configured to generate general mapping data from the base training data, the general mapping data being obtained by converting the training text into international phonetic alphabets and learning cross-language general mapping relationships at a phoneme level; a base model training module configured to train a base model based on the general mapping data, the base model being an automatic speech recognition model adopting a non-self-recurrent text encoder and a self-recurrent decoder architecture; a model fine-tuning module configured to fine-tune model parameters of the base model using the fine-tuning data; an online preference optimization module configured to perform online preference optimization on the fine-tuned base model according to a prompt set and a multi-objective reward function to obtain an adapted model, the prompt set including target language text and reference audio, and the multi-objective reward function being constructed based on automatic speech recognition, speaker verification, and perceptual evaluation speech quality.

9. A computer device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 7 when executing the computer program.

10. A storage medium having stored thereon a computer program, characterized in that The computer program implements the method of any one of claims 1 to 7 when executed by the processor.

Images