Speech synthesis and speech recognition

Abstract

According to the implementation of the present disclosure, a solution for speech synthesis and speech recognition is proposed. According to the solution, a text-to-speech (TTS) model and an automatic speech recognition (ASR) model supporting at least one language are acquired. The TTS model and the ASR model are adjusted based on a first set of paired data of a target language to support the target language. Then, the TTS model is optimized based on the first set of paired data and a first set of synthetic paired data of the target language generated by the ASR model, while the ASR model is optimized based on the first set of paired data and a second set of synthetic paired data of the target language generated by the TTS model. In this way, the solution can provide a TTS model and an ASR model with higher accuracy for languages ​​that lack training data using less training data.

Description

Background Art

[0001] Speech synthesis (also known as text-to-speech, TTS) and speech recognition (also known as automatic speech recognition, ASR) are important speech tasks and require a large amount of text and speech data pairs for model training. However, there are more than 6,000 languages ​​in the world. Most languages ​​lack speech training data, which makes it difficult to provide TTS and ASR functions for these languages ​​because large amounts of expensive speech training data are required to ensure the accuracy of the trained TTS and ASR models. Summary of the Invention

[0002] According to an implementation of the present disclosure, a solution for speech synthesis and speech recognition is proposed. According to this solution, a text-to-speech (TTS) model and an automatic speech recognition (ASR) model supporting at least one language are acquired. The TTS model and the ASR model are adjusted to support the target language based on a first set of paired data in a target language. The first set of paired data includes speech data in the target language from multiple speakers and their corresponding text data. The TTS model is then optimized based on the first set of paired data and a first set of synthetic paired data in the target language, while the ASR model is simultaneously optimized based on the first set of paired data and a second set of synthetic paired data in the target language. The first set of synthetic paired data includes a first set of speech data from multiple speakers and a first set of text data generated by the ASR model based on the first set of speech data, and the second set of synthetic paired data includes a second set of text data and a second set of speech data for multiple speakers generated by the TTS model based on the second set of text data. In this way, this solution can provide TTS models and ASR models with higher accuracy for languages ​​lacking training data using less training data.

[0003] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] FIG1 illustrates a schematic diagram of an example environment in which implementations of the present disclosure may be implemented;

[0005] Figure 2 A schematic block diagram illustrating the working principle of a model training device according to an implementation of the present disclosure is shown;

[0006] Figure 3A A schematic diagram showing an example model structure of a TTS model according to an implementation of the present disclosure;

[0007] Figure 3BA schematic diagram showing an example model structure of an ASR model according to an implementation of the present disclosure;

[0008] Figure 4 A flowchart illustrating an example method for training a TTS model and an ASR model according to an implementation of the present disclosure; and

[0009] Figure 5 A block diagram of an electronic device capable of implementing various implementations of the present disclosure is shown.

[0010] In these drawings, the same or similar reference symbols are used to designate the same or similar elements. DETAILED DESCRIPTION

[0011] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and implement the present disclosure, rather than to imply any limitation on the scope of the present disclosure.

[0012] As used herein, the term "including" and its variations are to be interpreted as open-ended terms meaning "including but not limited to." The term "based on" is to be interpreted as "based, at least in part, on." The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation." The term "another implementation" is to be interpreted as "at least one other implementation." The terms "first," "second," and so on may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0013] As mentioned above, speech synthesis (also known as text-to-speech, TTS) and speech recognition (also known as automatic speech recognition, ASR) are important speech tasks.

[0014] The goal of TTS is to synthesize understandable, natural speech from text. Training a TTS model typically requires high-quality recordings of a single speaker collected in a professional recording studio. To improve pronunciation accuracy, TTS models often require a pronunciation dictionary to convert character sequences into phoneme sequences as model input. This is known as grapheme-to-phoneme conversion. Furthermore, TTS models use text normalization rules to convert irregular words into more pronounceable, standardized text.

[0015] The goal of ASR is to generate correct text from a speech sequence. Training an ASR model typically requires speech data from multiple speakers, enabling the trained ASR model to recognize speech data from unknown speakers. The multiple-speaker speech data used to train an ASR model may not need to be as high-quality as the speech data used to train a TTS model, but the data volume is typically an order of magnitude larger. Alternatively, the ASR model can first recognize speech as a sequence of phonemes, and then further convert this into a sequence of characters using a phonetic dictionary, similar to a TTS model.

[0016] In traditional solutions, for languages ​​with abundant speech training data (also known as resource-rich languages), a large amount of paired speech and text data is often required to train TTS models and ASR models to achieve high accuracy. For example, training a TTS model typically requires dozens of hours of high-quality recordings of a single speaker and their corresponding text (also known as "single-speaker high-quality paired data"), while training an ASR model requires at least hundreds of hours of low-quality speech data of multiple speakers and their corresponding text (also known as "multi-speaker low-quality paired data"). In addition, for resource-rich languages, TTS models can also use pronunciation dictionaries to improve pronunciation accuracy. Optionally, unpaired speech data and unpaired text data can be used to improve the performance of TTS models and ASR models.

[0017] Traditionally, for languages ​​lacking speech training data (also known as resource-poor languages), high-quality, single-speaker paired data for training TTS models can be reduced to tens of minutes, while low-quality, multi-speaker paired data for training ASR models can be reduced to tens of hours. Furthermore, for resource-poor languages, unpaired speech data and unpaired text data are often used to improve the performance of TTS and ASR models.

[0018] As can be seen from the above description, traditional solutions often require a large amount of text and speech data pairs for model training to ensure the accuracy of TTS models and ASR models. However, there are more than 6,000 languages ​​in the world, and most languages ​​lack speech training data. Although the amount of training data can be reduced for resource-poor languages, collecting speech and text paired data for training TTS models and ASR models (for example, TTS models require tens of minutes, ASR models require tens of hours), pronunciation dictionaries, and a large amount of high-quality unpaired speech data of a single speaker still incurs high data collection costs. Moreover, in the case of resource shortages, the accuracy of TTS models and ASR models is generally low.

[0019] According to the implementation of the present disclosure, a solution for speech synthesis and speech recognition is proposed, which can provide a highly accurate TTS model and ASR model for resource-poor languages ​​at a lower data collection cost.

[0020] To achieve lower data collection costs, this approach uses only the following data to train the TTS and ASR models: 1) as little as possible (e.g., a few minutes) of high-quality, single-speaker paired data; 2) a small amount (e.g., several hours) of low-quality, multi-speaker paired data; and 3) a large amount (e.g., tens of hours) of low-quality, multi-speaker unpaired speech data. Compared to traditional approaches, this approach does not use single-speaker high-quality unpaired speech data or a pronunciation dictionary, but instead directly uses characters as input for the TTS model and output for the ASR model.

[0021] To achieve higher accuracy, the solution first pre-trains the TTS model and ASR model on multiple languages ​​with rich resources and a large amount of paired data to learn the alignment ability between speech and text. Then, the solution adjusts the TTS model and ASR model that support multiple languages ​​to the target language (for example, a language with scarce resources). The solution further uses unpaired speech data and unpaired text data to perform dual training of the TTS model and ASR model to iteratively improve the accuracy of each other. In addition, the solution uses the dual-trained TTS model and ASR model to synthesize paired data to train the target TTS model and target ASR model, so that the target TTS model can support high-quality speech of a single speaker and the accuracy of the target TTS model and target ASR model is further improved.

[0022] TTS and ASR can generally be formulated as sequence-to-sequence problems. Assume that a pair of text and speech sequences is represented as (x, y) ∈ D, where D represents the set of text and speech sequence pairs used to train the TTS model and the ASR model. Each element in the text sequence x represents a phoneme or character, and each element in the speech sequence y represents a speech unit (also called a "frame"). In some implementations, to train the TTS model θ, a mean square error loss function as shown below can be used:

[0023]

[0024] Where f(x;θ) represents the speech sequence corresponding to the text sequence x generated by the TTS model θ. In some implementations, to train the ASR model φ, the negative log-likelihood loss function shown below can be used:

[0025]

[0026] TTS models and ASR models can be trained based on the encoder-attention-decoder framework, where the encoder converts the source sequence into a set of hidden representations, and the decoder regressively generates the target sequence based on the source hidden representations obtained through the attention mechanism.

[0027] In the following, the paired data for training TTS models on resource-rich languages ​​is also denoted as D rich_tts , and the paired data for training ASR models on resource-rich languages ​​is also denoted as D rich_asr The single-speaker high-quality paired data for the target speaker is also denoted as D h , and the multi-speaker low-quality paired data is also denoted as D l Unpaired text data is also denoted as X u , and the multi-speaker low-quality unpaired speech data is also denoted as Y u .

[0028] Various implementations of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1A and Figure 1B 1 shows a schematic diagram of an example environment 100 in which implementations of the present disclosure may be implemented. Figure 1A As shown, the environment 100 may include a model training device 120 configured to train a TTS model 130 and an ASR model 140 based on a training dataset 110. Figure 1B As shown, the TTS model 130 can generate speech 132 corresponding to the text 131 based on the input text 131. The ASR model 140 can generate text 142 corresponding to the speech 141 based on the input speech 141. It should be understood that the structure and function of the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. The embodiments of the present disclosure can also be applied to environments with different structures and / or functions.

[0030] Figure 2 FIG. 1 is a schematic block diagram illustrating the working principle of the model training device 120 according to the implementation of the present disclosure. Figure 2 As shown, the model training device 120 may include a pre-training and adjustment module 121, a dual optimization module 122, and a knowledge distillation module 123. It should be understood that the structure and function of the model training device 120 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. In some implementations, the model training device 120 may be used in conjunction with Figure 2 Different structures to achieve.

[0031] like Figure 2As shown, the training data set 110 may include a set of paired data 111 (also referred to herein as "second set of paired data") of at least one source language. The paired data 111 may include multiple pairs of speech and text data of at least one source language, wherein the speech data may be speech data from multiple speakers. The pre-training and adjustment module 121 is configured to pre-train a TTS model and an ASR model that support at least one source language based on the second set of paired data 111. At least one source language may be a resource-rich language, such as English, Chinese, etc. The reasons for using group paired data of resource-rich languages ​​to pre-train the TTS model and the ASR model are: 1) In commercial voice services, group paired data for resource-rich languages ​​are often easy to obtain; 2) In addition, due to the pronunciation similarities between different human languages, the alignment capabilities of speech and text learned on resource-rich languages ​​can be transferred to resource-poor languages. In some implementations, the second set of paired data 111 may include D rich_tts and D rich_asr The pre-training and adjustment module 121 can be based on D according to the above formula (1). rich_tts To pre-train the TTS model θ, and according to the above formula (2) based on D rich_asr To pre-train the ASR model φ.

[0032] like Figure 2 As shown, the training dataset 110 may also include a set of paired data 112 in a target language (also referred to herein as a "first set of paired data"). The target language may be a language different from at least one source language. For example, the target language may be a language with limited resources. The first set of paired data 112 may include multiple pairs of speech and text data in the target language, where the speech data may be speech data from multiple speakers. The pre-training and adjustment module 121 is configured to adjust the pre-trained TTS model and ASR model based on the first set of paired data 112 to obtain a TTS model 131 and an ASR model 141 that support the target language.

[0033] The source language and the target language typically have different phoneme / character vocabularies and are associated with different speakers. In some implementations, the pre-training and tuning module 121 may initialize the TTS model on the target language by using all pre-trained parameters in the pre-trained TTS model except for the phoneme / character embedding and the speaker embedding. The pre-training and tuning module 121 may initialize the ASR model on the target language by using all pre-trained parameters in the pre-trained ASR model except for the phoneme / character embedding. In some implementations, the first set of paired data 112 may include single-speaker high-quality paired data D for the target speaker. h and multi-speaker low-quality paired data D lThe pre-training and adjustment module 121 can be based on D according to the above formula (1). h and D l Both are used to adjust the pre-trained TTS model θ to support the target language, and according to the above formula (2) based on D h and D l Both are used to adjust the pre-trained ASR model φ to support the target language.

[0034] TTS and ASR are two dual tasks, and their dual properties can be exploited to improve each other's accuracy, especially in resource-limited settings.

[0035] In some implementations, such as Figure 2 As shown, the dual optimization module 122 can utilize the dual training between TTS and ASR to improve the conversion ability between text and speech. Figure 2 As shown, the training data set 110 may include a non-paired speech and text data set 113 of the target language, wherein the non-paired speech data set is also referred to herein as a “first set of speech data”, which includes, for example, a plurality of Y u , and the non-paired text data set is also referred to herein as the "second set of text data", which for example includes a plurality of X u In some implementations, for each Y in the first set of speech data u , the dual optimization module 122 can use the ASR model 141 to generate corresponding text data. The dual optimization module 122 can combine the first set of speech data and the corresponding text data generated by the ASR model 141 based on the first set of speech data (also referred to as "first set of text data" in this article) into a first set of synthetic paired data, and train the TTS model θ according to the above formula (1). Since the ASR model can recognize the speech of multiple speakers, by using the ASR model to convert the speech from multiple speakers into corresponding text, it is helpful to train a TTS model that can synthesize the speech of multiple speakers. In addition, since the ASR model can recognize the speech of unknown speakers, using the synthesized paired data to train the TTS model can enable it to have the ability to synthesize the speech of unknown speakers. In some implementations, for each X in the second set of text data u , the dual optimization module 122 can use the TTS model 131 to generate corresponding speech data. For example, the dual optimization module 122 can randomly select a speaker identifier (ID) and uSynthesize the speaker's speech. The dual optimization module 122 can combine the second set of text data and the corresponding speech data generated by the TTS model 131 based on the second set of text data (also referred to herein as the "second set of speech data") into a second set of synthesized paired data, and train the ASR model φ according to the above formula (2). This helps train an ASR model capable of recognizing the speech of multiple speakers.

[0036] In some implementations, the dual optimization module 122 can optimize the TTS model and the ASR model using multiple iterations, where each iteration utilizes the latest TTS model and ASR model to generate new synthetic paired data for each other. Additionally or alternatively, in some implementations, in addition to utilizing the synthetic paired data, the dual optimization module 122 can also use the paired data 112 during the iterative optimization of the TTS model and the ASR model to ensure the accuracy of the trained TTS model and ASR model. For example, the dual optimization module 122 can use a synthetic paired data set based on unpaired data for training in one iteration and use the synthetic paired data 112 for training in the next iteration. For another example, the dual optimization module 122 can use the synthetic paired data and the synthetic paired data 112 in a predetermined ratio (e.g., 1:1) for training in each iteration. In this way, the dual optimization module 122 can obtain optimized TTS models 132 and ASR models 142 that support multiple speakers.

[0037] In some cases, the dual-trained TTS model 132 and ASR model 142 may not be able to meet the needs of online deployment for the following reasons: 1) although the TTS model 132 can support multiple speakers, the speech quality for the target speaker may not be good enough; 2) the speech synthesized by the TTS model 132 may have problems with word omissions and word duplication; 3) the accuracy of the ASR model 142 may need to be further improved.

[0038] In some implementations, such as Figure 2 As shown, the knowledge distillation module 123 can train the target TTS model 130 and the target ASR model 140 based on the first set of paired data 112 and the multiple sets of synthetic paired data 114 generated by the TTS model 132 and the ASR model 142, so that the target TTS model 130 can support high-quality speech of a single speaker and the accuracy of the target TTS model 130 and the target ASR model 140 can be further improved.

[0039] In some implementations, as described above, the paired data 112 may include single-speaker high-quality paired data D for the target speaker. h and multi-speaker low-quality paired data D lIn order to enable the target TTS model 130 to synthesize high-quality speech of the target speaker, the knowledge distillation module 123 can extract single-speaker high-quality paired data D for the target speaker from the paired data 112. h (also referred to herein as “the third set of paired data”) is used to train the target TTS model 130 .

[0040] In some implementations, the knowledge distillation module 123 may also utilize the TTS model 132 to synthesize a large amount of speech data of the target speaker (also referred to herein as the "third set of speech data") based on a large amount of unpaired text data (also referred to herein as the "third set of text data"). The third set of text data, for example, includes multiple X u In some implementations, for each X in the third set of text data u , the knowledge distillation module 123 can use the TTS model 132 to synthesize the speech data of the corresponding target speaker. For example, the knowledge distillation module 123 can input the text X to the TTS model 132 u and the ID of the target speaker, thereby synthesizing the speech of the target speaker. The knowledge distillation module 123 can combine the third group of text data and the third group of speech data into a third group of synthetic paired data for training the target TTS model. Additionally or alternatively, since there may be problems such as word omissions and word duplications in the third group of speech data synthesized in the TTS model 132, in some implementations, the knowledge distillation module 123 can filter out unqualified speech data in the third group of speech data and filter out text data corresponding to the unqualified speech data in the third group of text data to obtain a fourth group of synthetic paired data. The knowledge distillation module 123 can train the target TTS model 130 based on the third group of paired data extracted from the paired data 112 and the synthesized and filtered fourth group of synthetic paired data, and according to the above formula (1), so that the target TTS model 130 can synthesize high-quality speech of the corresponding target speaker based on the input text data of the target language.

[0041] In some implementations, the unqualified speech data filtered out by the knowledge distillation module 123 may include at least one of the following: speech data with word omissions; speech data with word repetitions; and unintelligible speech data. In some implementations, the knowledge distillation module 123 may filter out unqualified speech data from the third set of speech data based on at least one of the following two metrics: word coverage ratio (WCR) and attention diagonal ratio (ADR), where WCR is negatively correlated with the likelihood of word omissions or word repetitions in the speech data, and ADR indicates the degree of alignment between the speech data and the text data used to synthesize the speech data.

[0042] In some implementations, the knowledge distillation module 123 may determine the WCR of the synthesized speech data based on the following formula:

[0043]

[0044] Where N represents the number of words in the text corresponding to the synthesized speech data, T i A represents the number of characters in the i-th word, and S represents the number of speech units in the target sound spectrum corresponding to the synthesized speech. t，s represents the element in the tth row and sth column of the attention weight matrix A, where the attention weight matrix A can be determined based on the encoder-decoder attention weights in the TTS model 132. The WCR is negatively correlated with the likelihood of word omission or word repetition in the synthesized speech data. That is, a higher WCR indicates a lower likelihood of word omission or word repetition in the synthesized speech data. In some implementations, the knowledge distillation module 123 can filter out speech data with a WCR below a predetermined threshold from the third set of speech data.

[0045] In some implementations, the knowledge distillation module 123 may determine the ADR of the synthesized speech data based on the following formula:

[0046]

[0047] Where T represents the number of characters in the text corresponding to the synthesized speech data, S represents the number of speech units in the synthesized speech data, k = S / T, and b represents a hyperparameter used to determine the width of the diagonal line. A higher ADR indicates a better degree of attention alignment between the synthesized speech data and the corresponding text, and therefore a lower likelihood of problems such as word omissions, word repetitions, or speech collapse (incomprehensibility). In some implementations, the knowledge distillation module 123 can filter out speech data with an ADR below a predetermined threshold from the third set of speech data.

[0048] In some implementations, the knowledge distillation module 123 can utilize the TTS model 132 to synthesize speech data of multiple speakers (also referred to herein as the "fourth set of speech data") based on a large amount of unpaired text data (also referred to herein as the "third set of text data"). The third set of text data, for example, includes multiple X u In some implementations, for each X in the third set of text data u , the knowledge distillation module 123 can randomly select a speaker's identifier (ID) and uSynthesize the speaker's speech. The knowledge distillation module 123 can combine the third group of text data and the fourth group of speech data into a fifth group of synthetic paired data for training the target ASR model. Additionally or alternatively, in some implementations, the knowledge distillation module 123 can utilize the ASR model 142 to generate corresponding text data (also referred to herein as the "fifth group of speech data") based on a large amount of unpaired speech data (also referred to herein as the "fifth group of text data"). The knowledge distillation module 123 can combine the fifth group of speech data and the fourth group of text data into a sixth group of synthetic paired data for training the target ASR model. In some implementations, the knowledge distillation module 123 can generate corresponding text data (also referred to herein as the "fourth group of text data") based on the ASR model 142 based on the large amount of unpaired speech data (also referred to herein as the "fifth group of speech data"). h and multi-speaker low-quality paired data D l ), the fifth group of synthetic paired data and the sixth group of synthetic paired data, and train the target ASR model 140 according to the above formula (2) so that the target ASR model 140 can generate text data corresponding to the speech data based on the speech data of the target language from multiple speakers.

[0049] Figure 3A A block diagram illustrating an example model structure 210 of a TTS model according to implementations of the present disclosure is shown. Figure 3B FIG. 2 is a block diagram illustrating an example model structure 220 of an ASR model according to an implementation of the present disclosure. Figure 2 The TTS models 130, 131 and 132 shown may have Figure 3A As shown in the model structure 210, the ASR models 140, 141 and 142 may have the following Figure 3B The model structure 220 is shown. In some implementations, as Figure 3A and 3B As shown, both the TTS model and the ASR model can adopt a transformer-based encoder-attention-decoder structure. In some implementations, to better capture dependencies in long speech sequences, the feedforward network in the traditional transformer model can be replaced with a one-dimensional convolutional neural network to implement the TTS model and the ASR model.

[0050] like Figure 3AAs shown, the TTS model 210 may include a text input module 211, an encoder 212, a speaker module 213, a speech input module 214, a decoder 215, and an output module 216. In the TTS model 210, the input module 211 of the encoder 212 may be a character / phoneme embedding vector lookup table that converts character / phoneme IDs into embedding vectors. The input module 214 of the decoder 215 may be a speech pre-net layer that includes multiple dense layers to nonlinearly convert speech signals. The output module 216 of the decoder 215 includes a linear layer for converting hidden feature representations into a spectrum, and a stop linear layer that uses a sigmoid function to predict whether the current step should stop. The TTS model 210 that supports multiple speakers utilizes the speaker module 213 to distinguish multiple speakers. In some implementations, speaker embedding vectors may be added to both the output of the encoder 212 and the input of the decoder 215 (i.e., the output of the input module 214). Specifically, the speaker ID can be converted into a speaker embedding vector using an embedding vector lookup table, and then a linear transformation and a softsign function (e.g., x = x / (1+|x|)) are applied to the speaker embedding vector. The transformed result is then connected to the output of the encoder 212 or the input of the decoder 215, and another linear layer is used to reduce the hidden vector dimension to the original dimension of the output of the encoder 212 or the input of the decoder 215. The encoder 212 and the decoder 215 can have the same implementation as the encoder and decoder in the traditional scheme, and will not be repeated here.

[0051] like Figure 3B As shown, the ASR model 220 may include a speech input module 214, an encoder 222, a text input module 223, a decoder 224, and an output module 225. In the ASR model 220, the input module 221 of the encoder 222 may include multiple convolutional layers to reduce the length of the speech sequence. The input module 223 of the decoder 224 may be a character / phoneme embedding vector lookup table. The output module 225 of the decoder 224 may include a linear layer and a softmax function, wherein the linear layer shares the same weights with the character / phoneme embedding vector lookup table in the input module 223. The encoder 222 and the decoder 224 may have the same implementation as the encoder and decoder in the traditional solution, and will not be repeated here.

[0052] Figure 4 FIG. 4 is a flow chart showing an example method 400 for training a TTS model and an ASR model according to an implementation of the present disclosure. For example, the method 400 may be as shown in FIG. Figure 2 It should be understood that the method 400 may further include additional actions not shown and / or may omit actions shown, and the scope of the present disclosure is not limited in this respect.

[0053] like Figure 4 As shown, at box 410, the model training device 120 obtains a TTS model and an ASR model that support at least one language.

[0054] In some implementations, obtaining the TTS model and the ASR model includes training the TTS model and the ASR model based on a second set of paired data in the at least one language, wherein the second set of paired data includes speech data in the at least one language from multiple speakers and corresponding text data.

[0055] At block 420 , the model training device 120 adjusts the TTS model and the ASR model to support the target language based on a first set of paired data in the target language, wherein the first set of paired data includes speech data in the target language from a plurality of speakers and corresponding text data thereof.

[0056] In some implementations, the target language is different from the at least one language.

[0057] At box 430, the model training device 120 optimizes the TTS model based on the first set of paired data and a first set of synthetic paired data in the target language, and simultaneously optimizes the ASR model based on the first set of paired data and a second set of synthetic paired data in the target language, wherein the first set of synthetic paired data includes a first set of speech data from multiple speakers and a first set of text data generated by the ASR model based on the first set of speech data, and the second set of synthetic paired data includes a second set of text data and a second set of speech data for multiple speakers generated by the TTS model based on the second set of text data.

[0058] At block 440 , the model training device 120 trains a target TTS model and a target ASR model based on the first set of paired data and multiple sets of synthetic paired data in the target language generated by the optimized TTS model and ASR model.

[0059] In some implementations, training the target TTS model includes: obtaining a third set of paired data associated with a target speaker among the multiple speakers from the first set of paired data, wherein the third set of paired data includes speech data in the target language from the target speaker and its corresponding text data; using the optimized TTS model to generate a third set of synthetic paired data in the target language, wherein the third set of synthetic paired data includes a third set of text data and a third set of speech data of the target speaker generated by the optimized TTS model based on the third set of text data; and training the target TTS model based on the third set of paired data and the third set of synthetic paired data, so that the target TTS model can generate speech data of the target speaker corresponding to the text data based on the text data in the target language.

[0060] In some implementations, training the target TTS model based on the third set of paired data and the third set of synthetic paired data includes: filtering unqualified voice data in the third set of voice data and text data corresponding to the unqualified voice data in the third set of text data to obtain a fourth set of synthetic paired data; and training the target TTS model based on the third set of paired data and the fourth set of synthetic paired data.

[0061] In some implementations, the unqualified speech data includes at least one of the following: speech data with word omissions; speech data with word repetitions; and unintelligible speech data.

[0062] In some implementations, filtering the unqualified speech data includes filtering speech data in the third set of speech data whose word coverage rate (WCR) is lower than a predetermined threshold, wherein the WCR is negatively correlated with the possibility of word omission or word repetition in the speech data.

[0063] In some implementations, filtering the unqualified speech data includes filtering speech data in the third set of speech data whose attention diagonal ratio (ADR) is lower than a predetermined threshold, wherein the ADR indicates the degree of alignment between the speech data and text data in the third set of text data used to generate the speech data.

[0064] In some implementations, training the target ASR model includes: generating a fifth set of synthetic paired data in the target language using the optimized TTS model, the fifth set of synthetic paired data including a third set of text data and a fourth set of speech data from multiple speakers generated by the optimized TTS model based on the third set of text data; generating a sixth set of synthetic paired data in the target language using the optimized ASR model, wherein the sixth set of synthetic paired data includes a fifth set of speech data from multiple speakers and a fourth set of text data generated by the optimized ASR model based on the fifth set of speech data; and training the target ASR model based on the first set of paired data, the fifth set of synthetic paired data, and the sixth set of synthetic paired data, so that the target ASR model can generate text data corresponding to the speech data based on the speech data in the target language from multiple speakers.

[0065] As can be seen from the above description, the implementation of the present disclosure proposes a solution for speech synthesis and speech recognition that can provide a TTS model and ASR model with high accuracy for resource-poor languages ​​at a low data collection cost.

[0066] To achieve lower data collection costs, this solution uses only the following data to train the TTS and ASR models: 1) as little as possible (e.g., a few minutes) of high-quality, single-speaker paired data; 2) a small amount (e.g., several hours) of low-quality, multi-speaker paired data; and 3) a large amount (e.g., tens of hours) of low-quality, multi-speaker unpaired speech data. Compared to traditional approaches, this solution does not use single-speaker high-quality unpaired speech data and does not utilize a pronunciation dictionary, significantly reducing the data collection costs used to train the TTS and ASR models.

[0067] To achieve higher accuracy, the solution first pre-trains the TTS model and ASR model on multiple languages ​​with rich resources and a large amount of paired data to learn the alignment ability between speech and text. Then, the solution adjusts the TTS model and ASR model that support multiple languages ​​to the target language (for example, a language with scarce resources). The solution further uses unpaired speech data and unpaired text data to perform dual training of the TTS model and ASR model to iteratively improve the accuracy of each other. In addition, the solution uses the dual-trained TTS model and ASR model to synthesize paired data to train the target TTS model and target ASR model, so that the target TTS model can support high-quality speech of a single speaker and the accuracy of the target TTS model and target ASR model is further improved.

[0068] Figure 5FIG. 5 is a block diagram of an electronic device 500 capable of implementing various implementations of the present disclosure. Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described in the present disclosure. Figure 2 The model training device 120 shown can be implemented by the electronic device 500. Figure 5 As shown, electronic device 500 comprises an electronic device in the form of a general purpose computing device 500. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560.

[0069] In some implementations, the electronic device 500 can be implemented as various user terminals or service terminals. The service terminal can be a server, a large computing device, etc. provided by various service providers. The user terminal is such as a mobile terminal, a fixed terminal, or a portable terminal of any type, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that the electronic device 500 can support any type of interface for the user (such as a "wearable" circuit, etc.).

[0070] The processing unit 510 may be a real or virtual processor and may be capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of the electronic device 500. The processing unit 510 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0071] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any available media that is accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof.

[0072] The storage device 530 may be a removable or non-removable medium and may include a machine-readable medium such as a memory, a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 500. The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not described herein, Figure 5 As shown in FIG, a magnetic disk drive for reading or writing from a removable, nonvolatile magnetic disk and an optical disk drive for reading or writing from a removable, nonvolatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces.

[0073] The communication unit 540 enables communication with another computing device via a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0074] Input device 550 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 560 may be one or more output devices, such as a display, speaker, printer, etc. Electronic device 500 may also communicate with one or more external devices (not shown) via communication unit 540 as needed, such as storage devices, display devices, etc., with one or more devices that allow a user to interact with electronic device 500, or with any device that allows electronic device 500 to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0075] In some implementations, in addition to being integrated on a single device, some or all of the various components of the electronic device 500 may also be configured in the form of a cloud computing architecture. In a cloud computing architecture, these components can be remotely located and can work together to implement the functionality described in this disclosure. In some implementations, cloud computing provides computing, software, data access, and storage services that do not require the end user to be aware of the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing uses appropriate protocols to provide services over a wide area network (such as the Internet). For example, a cloud computing provider provides applications over a wide area network, and they can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data can be stored on servers in a remote location. The computing resources in a cloud computing environment can be consolidated at a remote data center location or they can be dispersed. The cloud computing infrastructure can provide services through a shared data center, even if they appear to be a single access point for users. Therefore, the components and functionality described herein can be provided by a service provider in a remote location using a cloud computing architecture. Alternatively, they can be provided from a conventional server, or they can be installed directly or otherwise on the client device.

[0076] The electronic device 500 can be used to implement video encoding in various implementations of the present disclosure. The memory 520 can include a video encoding module 522 having one or more program instructions that can be accessed and executed by the processing unit 510 to implement the functions of various implementations described herein.

[0077] Some example implementations of the present disclosure are listed below.

[0078] In a first aspect, the present disclosure provides a computer-implemented method. The method includes: obtaining a text-to-speech (TTS) model and an automatic speech recognition (ASR) model that support at least one language; adjusting the TTS model and the ASR model to support the target language based on a first set of paired data in a target language, wherein the first set of paired data includes speech data in the target language from multiple speakers and corresponding text data; and optimizing the TTS model based on the first set of paired data and a first set of synthetic paired data in the target language, and optimizing the ASR model based on the first set of paired data and a second set of synthetic paired data in the target language, wherein the first set of synthetic paired data includes a first set of speech data from multiple speakers and a first set of text data generated by the ASR model based on the first set of speech data, and the second set of synthetic paired data includes a second set of text data and a second set of speech data for multiple speakers generated by the TTS model based on the second set of text data.

[0079] In some implementations, obtaining the TTS model and the ASR model includes training the TTS model and the ASR model based on a second set of paired data in the at least one language, wherein the second set of paired data includes speech data in the at least one language from multiple speakers and corresponding text data.

[0080] In some implementations, the method further includes training a target TTS model and a target ASR model based on the first set of paired data and multiple sets of synthetic paired data in the target language generated by the optimized TTS model and ASR model.

[0081] In some implementations, training the target TTS model includes: obtaining a third set of paired data associated with a target speaker among the multiple speakers from the first set of paired data, wherein the third set of paired data includes speech data in the target language from the target speaker and its corresponding text data; using the optimized TTS model to generate a third set of synthetic paired data in the target language, wherein the third set of synthetic paired data includes a third set of text data and a third set of speech data of the target speaker generated by the optimized TTS model based on the third set of text data; and training the target TTS model based on the third set of paired data and the third set of synthetic paired data, so that the target TTS model can generate speech data of the target speaker corresponding to the text data based on the text data in the target language.

[0082] In some implementations, training the target TTS model based on the third set of paired data and the third set of synthetic paired data includes: filtering unqualified voice data in the third set of voice data and text data corresponding to the unqualified voice data in the third set of text data to obtain a fourth set of synthetic paired data; and training the target TTS model based on the third set of paired data and the fourth set of synthetic paired data.

[0083] In some implementations, the unqualified speech data includes at least one of the following: speech data with word omissions; speech data with word repetitions; and unintelligible speech data.

[0084] In some implementations, filtering the unqualified speech data includes filtering speech data in the third set of speech data whose word coverage rate (WCR) is lower than a predetermined threshold, wherein the WCR is negatively correlated with the possibility of word omission or word repetition in the speech data.

[0085] In some implementations, filtering the unqualified speech data includes filtering speech data in the third set of speech data whose attention diagonal ratio (ADR) is lower than a predetermined threshold, wherein the ADR indicates the degree of alignment between the speech data and text data in the third set of text data used to generate the speech data.

[0086] In some implementations, training the target ASR model includes: generating a fifth set of synthetic paired data in the target language using the optimized TTS model, wherein the fifth set of synthetic paired data includes a third set of text data and a fourth set of speech data from multiple speakers generated by the optimized TTS model based on the third set of text data; generating a sixth set of synthetic paired data in the target language using the optimized ASR model, wherein the sixth set of synthetic paired data includes a fifth set of speech data from multiple speakers and a fourth set of text data generated by the optimized ASR model based on the fifth set of speech data; and training the target ASR model based on the first set of paired data, the fifth set of synthetic paired data, and the sixth set of synthetic paired data, so that the target ASR model can generate text data corresponding to the speech data based on the speech data in the target language from multiple speakers.

[0087] In some implementations, the target language is different from the at least one language.

[0088] In a second aspect, the present disclosure provides an electronic device. The electronic device includes: a processing unit; and a memory coupled to the processing unit and containing instructions stored thereon, wherein the instructions, when executed by the processing unit, cause the electronic device to perform actions, the actions including: obtaining a text-to-speech (TTS) model and an automatic speech recognition (ASR) model that support at least one language; adjusting the TTS model and the ASR model to support the target language based on a first set of paired data in a target language, wherein the first set of paired data includes speech data in the target language from multiple speakers and corresponding text data thereof; and optimizing the TTS model based on the first set of paired data and a first set of synthetic paired data in the target language, and optimizing the ASR model based on the first set of paired data and a second set of synthetic paired data in the target language, wherein the first set of synthetic paired data includes a first set of speech data from multiple speakers and a first set of text data generated by the ASR model based on the first set of speech data, and the second set of synthetic paired data includes a second set of text data and a second set of speech data for multiple speakers generated by the TTS model based on the second set of text data.

[0089] In some implementations, obtaining the TTS model and the ASR model includes training the TTS model and the ASR model based on a second set of paired data in the at least one language, wherein the second set of paired data includes speech data in the at least one language from multiple speakers and corresponding text data.

[0090] In some implementations, the actions further include: training a target TTS model and a target ASR model based on the first set of paired data and multiple sets of synthetic paired data in the target language generated by the optimized TTS model and ASR model.

[0091] In some implementations, training the target TTS model includes: obtaining a third set of paired data associated with a target speaker among the multiple speakers from the first set of paired data, wherein the third set of paired data includes speech data in the target language from the target speaker and its corresponding text data; using the optimized TTS model to generate a third set of synthetic paired data in the target language, wherein the third set of synthetic paired data includes a third set of text data and a third set of speech data of the target speaker generated by the optimized TTS model based on the third set of text data; and training the target TTS model based on the third set of paired data and the third set of synthetic paired data, so that the target TTS model can generate speech data of the target speaker corresponding to the text data based on the text data in the target language.

[0092] In some implementations, training the target TTS model based on the third set of paired data and the third set of synthetic paired data includes: filtering unqualified voice data in the third set of voice data and text data corresponding to the unqualified voice data in the third set of text data to obtain a fourth set of synthetic paired data; and training the target TTS model based on the third set of paired data and the fourth set of synthetic paired data.

[0093] In some implementations, the unqualified speech data includes at least one of the following: speech data with word omissions; speech data with word repetitions; and unintelligible speech data.

[0094] In some implementations, filtering the unqualified speech data includes filtering speech data in the third set of speech data whose word coverage rate (WCR) is lower than a predetermined threshold, wherein the WCR is negatively correlated with the possibility of word omission or word repetition in the speech data.

[0095] In some implementations, filtering the unqualified speech data includes filtering speech data in the third set of speech data whose attention diagonal ratio (ADR) is lower than a predetermined threshold, wherein the ADR indicates the degree of alignment between the speech data and text data in the third set of text data used to generate the speech data.

[0096] In some implementations, training the target ASR model includes: generating a fifth set of synthetic paired data in the target language using the optimized TTS model, wherein the fifth set of synthetic paired data includes a third set of text data and a fourth set of speech data from multiple speakers generated by the optimized TTS model based on the third set of text data; generating a sixth set of synthetic paired data in the target language using the optimized ASR model, wherein the sixth set of synthetic paired data includes a fifth set of speech data from multiple speakers and a fourth set of text data generated by the optimized ASR model based on the fifth set of speech data; and training the target ASR model based on the first set of paired data, the fifth set of synthetic paired data, and the sixth set of synthetic paired data, so that the target ASR model can generate text data corresponding to the speech data based on the speech data in the target language from multiple speakers.

[0097] In some implementations, the target language is different from the at least one language.

[0098] In a third aspect, the present disclosure provides a computer program product, which is tangibly stored in a non-transitory computer storage medium and includes machine-executable instructions, which, when executed by a device, cause the device to perform the method according to the first aspect.

[0099] In a fourth aspect, the present disclosure provides a computer-readable medium having machine-executable instructions stored thereon. When the machine-executable instructions are executed by a device, the device may perform the method according to the first aspect.

[0100] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), and the like.

[0101] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0104] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: Obtain a TTS model and an ASR model that supports at least one language. adapting the TTS model and the ASR model to support a target language based on a first set of paired data in the target language, wherein the first set of paired data includes speech data in the target language from a plurality of speakers and corresponding text data thereof, wherein the target language is different from the at least one language; as well as The TTS model is optimized based on the first set of paired data and a first set of synthetic paired data in the target language, and the ASR model is optimized based on the first set of paired data and a second set of synthetic paired data in the target language, wherein the first set of synthetic paired data includes a first set of speech data from multiple speakers and a first set of text data generated by the ASR model based on the first set of speech data, and the second set of synthetic paired data includes a second set of text data and a second set of speech data for multiple speakers generated by the TTS model based on the second set of text data.

2. The method according to claim 1, wherein obtaining the TTS model and the ASR model comprises: The TTS model and the ASR model are trained based on a second set of paired data for the at least one language, wherein the second set of paired data includes speech data and corresponding text data for the at least one language from a plurality of speakers.

3. The method according to claim 1, further comprising: Training a target TTS model and a target ASR model based on the first set of paired data and multiple sets of synthetic paired data of the target language generated by the optimized TTS model and the ASR model, wherein training the target TTS model includes: obtaining a third set of paired data associated with a target speaker from the plurality of speakers from the first set of paired data, wherein the third set of paired data includes speech data in the target language from the target speaker and corresponding text data; generating a third set of synthetic paired data in the target language using the optimized TTS model, wherein the third set of synthetic paired data includes a third set of text data and a third set of speech data of the target speaker generated by the optimized TTS model based on the third set of text data; and training a target TTS model based on the third set of paired data and the third set of synthetic paired data, so that the target TTS model can generate speech data of the target speaker corresponding to the text data based on the text data of the target language, and / or Wherein training the target ASR model comprises: generating a fifth set of synthetic paired data in the target language using the optimized TTS model, wherein the fifth set of synthetic paired data includes a third set of text data and a fourth set of speech data of a plurality of speakers generated by the optimized TTS model based on the third set of text data; generating a sixth set of synthetic paired data in the target language using the optimized ASR model, wherein the sixth set of synthetic paired data includes a fifth set of speech data from a plurality of speakers and a fourth set of text data generated by the optimized ASR model based on the fifth set of speech data; and A target ASR model is trained based on the first set of paired data, the fifth set of synthetic paired data, and the sixth set of synthetic paired data, so that the target ASR model can generate text data corresponding to the speech data based on the speech data in the target language from multiple speakers.

4. The method according to claim 3, wherein training the target TTS model based on the third set of paired data and the third set of synthetic paired data comprises: filtering unqualified voice data in the third set of voice data and text data corresponding to the unqualified voice data in the third set of text data to obtain a fourth set of synthetic paired data; as well as The target TTS model is trained based on the third set of paired data and the fourth set of synthetic paired data.

5. The method according to claim 4, wherein the unqualified voice data comprises at least one of the following: Speech data with word omissions; Speech data containing repeated words; and Unintelligible voice data.

6. The method according to claim 4, wherein filtering the unqualified voice data comprises: Speech data having a word coverage rate (WCR) lower than a predetermined threshold in the third group of speech data is filtered, wherein the WCR is negatively correlated with a probability of word omission or word repetition in the speech data.

7. The method according to claim 4, wherein filtering the unqualified voice data comprises: Speech data having an attention diagonal ratio (ADR) lower than a predetermined threshold in the third set of speech data is filtered, wherein the ADR indicates a degree of alignment between the speech data and text data in the third set of text data used to generate the speech data.

8. An electronic device comprising: processing unit; as well as a memory coupled to the processing unit and containing instructions stored thereon, wherein when the instructions are executed by the processing unit, the electronic device performs actions, the actions comprising: Obtain a TTS model and an ASR model that supports at least one language. adapting the TTS model and the ASR model to support a target language based on a first set of paired data in the target language, wherein the first set of paired data includes speech data in the target language from a plurality of speakers and corresponding text data thereof, wherein the target language is different from the at least one language; and The TTS model is optimized based on the first set of paired data and a first set of synthetic paired data in the target language, and the ASR model is optimized based on the first set of paired data and a second set of synthetic paired data in the target language, wherein the first set of synthetic paired data includes a first set of speech data from multiple speakers and a first set of text data generated by the ASR model based on the first set of speech data, and the second set of synthetic paired data includes a second set of text data and a second set of speech data for multiple speakers generated by the TTS model based on the second set of text data.

9. The electronic device according to claim 8, wherein acquiring the TTS model and the ASR model comprises: The TTS model and the ASR model are trained based on a second set of paired data for the at least one language, wherein the second set of paired data includes speech data and corresponding text data for the at least one language from a plurality of speakers.

10. The electronic device according to claim 8, wherein the actions further comprise: Training a target TTS model and a target ASR model based on the first set of paired data and multiple sets of synthetic paired data of the target language generated by the optimized TTS model and the ASR model, wherein training the target TTS model includes: obtaining a third set of paired data associated with a target speaker from the plurality of speakers from the first set of paired data, wherein the third set of paired data includes speech data in the target language from the target speaker and corresponding text data; generating a third set of synthetic paired data in the target language using the optimized TTS model, wherein the third set of synthetic paired data includes a third set of text data and a third set of speech data of the target speaker generated by the optimized TTS model based on the third set of text data; and training a target TTS model based on the third set of paired data and the third set of synthetic paired data, so that the target TTS model can generate speech data of the target speaker corresponding to the text data based on the text data of the target language, and / or Wherein training the target ASR model comprises: generating a fifth set of synthetic paired data in the target language using the optimized TTS model, wherein the fifth set of synthetic paired data includes a third set of text data and a fourth set of speech data of a plurality of speakers generated by the optimized TTS model based on the third set of text data; generating a sixth set of synthetic paired data in the target language using the optimized ASR model, wherein the sixth set of synthetic paired data includes a fifth set of speech data from a plurality of speakers and a fourth set of text data generated by the optimized ASR model based on the fifth set of speech data; and A target ASR model is trained based on the first set of paired data, the fifth set of synthetic paired data, and the sixth set of synthetic paired data, so that the target ASR model can generate text data corresponding to the speech data based on the speech data in the target language from multiple speakers.

11. The electronic device according to claim 10, wherein training the target TTS model based on the third set of paired data and the third set of synthetic paired data comprises: filtering unqualified voice data in the third set of voice data and text data corresponding to the unqualified voice data in the third set of text data to obtain a fourth set of synthetic paired data; as well as The target TTS model is trained based on the third set of paired data and the fourth set of synthetic paired data.

12. The electronic device according to claim 11, wherein the unqualified voice data comprises at least one of the following: Speech data with word omissions; Speech data containing repeated words; and Unintelligible voice data.

13. The electronic device according to claim 11, wherein filtering the unqualified voice data comprises: Speech data having a word coverage rate (WCR) lower than a predetermined threshold in the third group of speech data is filtered, wherein the WCR is negatively correlated with a probability of word omission or word repetition in the speech data.

14. The electronic device according to claim 11, wherein filtering the unqualified voice data comprises: Speech data having an attention diagonal ratio (ADR) lower than a predetermined threshold in the third set of speech data is filtered, wherein the ADR indicates a degree of alignment between the speech data and text data in the third set of text data used to generate the speech data.

15. A computer program product tangibly stored in a non-transitory computer storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform actions comprising: Obtain a TTS model and an ASR model that supports at least one language. adapting the TTS model and the ASR model to support a target language based on a first set of paired data in the target language, wherein the first set of paired data includes speech data in the target language from a plurality of speakers and corresponding text data thereof, wherein the target language is different from the at least one language; as well as The TTS model is optimized based on the first set of paired data and a first set of synthetic paired data in the target language, and the ASR model is optimized based on the first set of paired data and a second set of synthetic paired data in the target language, wherein the first set of synthetic paired data includes a first set of speech data from multiple speakers and a first set of text data generated by the ASR model based on the first set of speech data, and the second set of synthetic paired data includes a second set of text data and a second set of speech data for multiple speakers generated by the TTS model based on the second set of text data.

Images