Phoneme-based speech domain transfer method, system, and electronic device
By employing a phoneme-level speech domain transfer method, highly diverse speech segments are generated using a phoneme N-gram dictionary. This addresses the issues of high computational resource consumption and insufficient speaker diversity in existing technologies, achieving efficient domain transfer for speech recognition models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AISPEECH CO LTD
- Filing Date
- 2022-12-22
- Publication Date
- 2026-07-07
AI Technical Summary
In existing technologies, domain transfer based on neural network speech synthesis models requires a large amount of computing resources and has poor speaker diversity. In contrast, domain transfer based on word-guided speech splicing synthesis cannot effectively model word-to-word linking, resulting in insufficient speech segment diversity.
By converting target domain text from characters to phonemes, generating multiple phoneme N-gram sequences using a phoneme N-gram dictionary, speech segments with speaker diversity are generated in the target domain. Synthetic audio in the target domain is then synthesized by splicing these segments together, and hierarchical coding distance regularization is used to prevent overfitting.
It reduces the computational resource requirements, increases the speaker diversity of synthesized speech, effectively helps speech recognition models complete domain transfer, and avoids overfitting of ASR models on synthesized data.
Smart Images
Figure CN115985290B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent speech, and more particularly to a phoneme-based speech domain transfer method, system, and electronic device. Background Technology
[0002] Due to the holistic architecture of E2E (end-to-end) ASR (automatic speech recognition) models and the requirement for training with paired speech-text data, domain transfer has always been a challenging task.
[0003] Since unpaired text data is easier to collect than paired speech-text data, domain adaptation using large amounts of unpaired text is more practical in real-world scenarios. Thanks to the modular design of E2E, it can be customized using plain text data, but the improvement in E2E ASR training capabilities using unpaired text data is limited. To address this shortcoming, the following methods are used to customize E2E ASR models: domain transfer based on neural network speech synthesis models and domain transfer based on word-guided speech concatenation synthesis.
[0004] 1. Domain transfer based on neural network speech synthesis model: This method uses a large amount of single-speaker or multi-speaker speech data to train the speech synthesis model, uses target domain text and the synthesis model to generate target domain speech data, and trains the speech recognition model to achieve domain transfer.
[0005] 2. Domain transfer based on word-guided speech splicing synthesis: It utilizes an improved RNN-T to obtain word-level speech-text alignment information, uses target domain text and speech segments to splice together target domain speech data, and trains a speech recognition model to achieve domain transfer.
[0006] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:
[0007] The shortcomings of neural network-based speech synthesis models stem from the complexity of the neural network training and inference structure itself. A large amount of computing resources are required in the training and generation stages of the speech synthesis model, and the speaker diversity of the generated speech is poor due to the limited number of speakers in the training corpus.
[0008] Domain transfer based on word-guided speech splicing synthesis suffers from inherent limitations. The synthesized speech cannot model word-to-word linking, and the corresponding word segments do not contain linking information, resulting in poor diversity of word-level speech segments. Summary of the Invention
[0009] To at least address the problems of existing domain transfer methods requiring large amounts of computational resources and having poor speaker diversity, this invention provides a phoneme-based speech domain transfer method, comprising:
[0010] The target domain text is converted from character to phoneme to obtain the target domain phoneme sequence;
[0011] The target domain phoneme sequence is transformed into multiple phoneme N-gram sequences in the target domain according to the phoneme N-gram dictionary, wherein the phoneme N-gram dictionary is constructed from real speech in the source domain;
[0012] The target domain speech segments with speaker diversity are generated using multiple phoneme N-gram sequences in the target domain, and the target domain synthesized audio is generated based on the target domain speech segments.
[0013] Secondly, embodiments of the present invention provide a phoneme-based speech domain transfer system, comprising:
[0014] The phoneme sequence determination module is used to convert character-to-phoneme text in the target domain to obtain the target domain phoneme sequence.
[0015] The N-gram sequence conversion program module is used to convert the target domain phoneme sequence into multiple phoneme N-gram sequences in the target domain according to the phoneme N-gram dictionary, wherein the phoneme N-gram dictionary is constructed from real speech in the source domain;
[0016] The domain transfer procedure module is used to generate speaker-diverse speech segments in the target domain using multiple phoneme N-gram sequences in the target domain, and to generate synthesized audio in the target domain based on the speech segments in the target domain.
[0017] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the phoneme-based speech domain migration method of any embodiment of the present invention.
[0018] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the phoneme-based speech domain migration method of any embodiment of the present invention.
[0019] The beneficial effects of this invention are as follows: It generates speech for text in the target domain from a dictionary constructed using basic phoneme n-gram speech fragments. The phoneme-guided concatenation synthesis method has the ability to model word linking. Furthermore, since the concatenated audio fragments originate from a large amount of real speech, the synthesized speech exhibits better speaker diversity and reduces the required computational resources. This effectively helps train speech recognition models to complete domain transfer and avoids overfitting of ASR models on synthesized data. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a flowchart of a phoneme-based speech domain transfer method provided in an embodiment of the present invention;
[0022] Figure 2 This is a schematic diagram of splicing data generation for a phoneme-based speech domain transfer method according to an embodiment of the present invention;
[0023] Figure 3 This is a schematic diagram of phoneme-level SDG conversion for a phoneme-based speech domain transfer method provided in an embodiment of the present invention;
[0024] Figure 4 This is a schematic diagram of the duration of the GIGASPEECH target domain in a phoneme-based speech domain transfer method provided in an embodiment of the present invention;
[0025] Figure 5 This is a schematic diagram comparing word error rates under different settings of a phoneme-based speech domain transfer method according to an embodiment of the present invention;
[0026] Figure 6 This is a schematic diagram comparing word error rates with different weights α in the hierarchical coding distance regularization of a phoneme-based speech domain transfer method according to an embodiment of the present invention.
[0027] Figure 7 This is a schematic diagram of the structure of a phoneme-based speech domain transfer system provided in an embodiment of the present invention;
[0028] Figure 8 This is a schematic diagram of an embodiment of an electronic device for speech domain migration based on phonemes, provided as an example of the present invention. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0030] like Figure 1 The diagram shows a flowchart of a phoneme-based speech domain transfer method according to an embodiment of the present invention, which includes the following steps:
[0031] S11: Convert the target domain text from character to phoneme to obtain the target domain phoneme sequence;
[0032] S12: Convert the target domain phoneme sequence into multiple target domain phoneme N-gram sequences according to the phoneme N-gram dictionary, wherein the phoneme N-gram dictionary is constructed from real speech in the source domain;
[0033] S13: Generate speaker-diverse speech segments in the target domain using multiple phoneme N-gram sequences in the target domain, and generate synthesized audio in the target domain based on the speech segments in the target domain.
[0034] In this embodiment, in order to address the lack of fluency caused by word SDG (splicing data generation) during domain migration and to increase the diversity of synthesized speech, this method designs a phoneme-level SDG. Furthermore, to mitigate overfitting of the synthesized speech, layer-by-layer distance regularization is introduced between the speech codes generated by the adaptive and non-adaptive models.
[0035] For step S11, obtaining the target domain text is generally relatively easy, readily available through web scraping or input from those familiar with the field. After obtaining the target domain text, G2P (grapheme-to-phoneme) conversion is performed. Specifically, this involves querying a dictionary to convert the word sequence in the target domain text into a phoneme sequence. For words with multiple entries in the dictionary (i.e., synonyms), one entry is randomly selected during each synthesis process. Word boundaries are preserved and marked with \b to prepare for subsequent steps. The target domain phoneme sequence after G2P is as follows: Figure 2 The target domain phoneme sequence is shown in the annotation.
[0036] For step S12, the target domain phoneme sequence determined in step S11 is transformed into multiple phoneme N-gram sequences using a phoneme N-gram dictionary. This method requires pre-construction of the phoneme N-gram dictionary, which includes generating the dictionary based on the forced alignment results of the source domain's real speech. Specifically, by performing FA (Forced Alignment) on existing source domain real speech, where FA refers to the process of determining the start and end positions of each word (phoneme) given audio and text, the position of each phoneme in the original audio can be obtained. Furthermore, complete sentences in the source domain's real speech are divided into phoneme-level data, and a phoneme N-gram dictionary is constructed using this data, which is a one-to-many mapping from phoneme n-grams to corresponding speech segments.
[0037] The constructed phoneme N-gram dictionary is used to transform a target domain phoneme sequence into multiple target domain phoneme N-gram sequences. During the transformation process, a forced alignment is achieved by using a 3≤n≤10 gradation method. The phoneme N-gram dictionary is denoted as P, and its key set is denoted as S. For example... Figure 2 As shown in "deconstructing into multiple phonemes n-grams", the proposed pipeline is completed in 3 stages.
[0038] As one implementation method, this method adds silence sequences to both ends of each phoneme N-gram sequence to simulate pauses in real speech dialogue.
[0039] In this embodiment, word boundaries are preserved and marked with \b in the preceding steps for subsequent silence insertion. Specifically, in the forced alignment results of actual speech data, there are occasional silences between words, while continuous silences exist at both ends of all utterances. To simulate real speech data, this method also adds silences to both ends of the phoneme sequence. Furthermore, based on statistics in the forced alignment results, word boundaries are randomly removed or replaced with silences. The phoneme sequence generated after random silence insertion is as follows: Figure 2 As shown in "Randomly Insert Silent Fragments".
[0040] After appending silence sequences to both ends of each phoneme N-gram sequence, the transformation process further includes searching the phoneme N-gram dictionary P to find the phoneme n-gram with the maximum average length (minimum number of phoneme n-grams). This is implemented using a greedy algorithm that operates in a divide-and-conquer manner.
[0041] The algorithm is as follows:
[0042] Input: x, the phoneme sequence
[0043] Output: y, the list of disassembled n-gram sequences.
[0044] Requirement: S, the set of all phoneme n-grams in the dictionary P.
[0045]
[0046]
[0047] Here, the symbol × denotes the Cartesian product. It can be proven through mathematical induction that if such a sequence exists, the above algorithm will always return a sequence consisting of the minimum number of phoneme n-grams. This preserves the fluency of the synthesized audio well. If more decomposed sequences are returned, the first 10 decomposed sequences are taken, and the input sequences that cannot be decomposed using this process are discarded.
[0048] For step S13, the phoneme n-gram is converted into an actual speech segment by randomly selecting one of the speech segments corresponding to each phoneme n-gram from the phoneme N-gram dictionary P. Then, the speech segments are concatenated into a complete speech. For example... Figure 3 The example shown converts target domain text into a target domain phoneme N-gram sequence, reducing the number of tokens from 7 (i.e., the number of words in the SDG) to 5, which also improves fluency. Since the concatenated speech segments are derived from a large amount of real speech, the synthesized speech exhibits better speaker diversity.
[0049] As one implementation, after generating synthetic audio in the target domain, the method further includes: training an automatic speech recognition model using the synthetic audio in the target domain and real speech in the source domain;
[0050] Specifically, when training with real speech from the source domain, the automatic speech recognition loss of the automatic speech recognition model is subjected to distance regularization.
[0051] In this implementation, encoder weight freezing is typically used to prevent the ASR model from overfitting to synthesized speech. However, adapting a model with a weight-frozen encoder to a new target domain becomes more challenging due to the reduction in trainable parameters. In this method, the encoder is not frozen.
[0052] Instead, LEDR (Layer-wise Encoding Distance Regularization) is added to the ASR loss function. This term is similar to that used for each real speech sample x. The regularization term of the ASR loss penalizes the adaptive model φ of the l-th layer. l (x) and the unadaptive model φ′ l (x) represents the L1 distance between the normalized real speech codes generated. Here, the model speech codes are derived from clean-noise speech pairs encoded by the same model, and the loss of the speech codes is generated by adaptive and non-adaptive models with the same real speech input:
[0053]
[0054] Where L is the total number of encoder layers in the ASR model, and θ and θ' are the parameters of the adaptive and non-adaptive models in the ASR model, respectively.
[0055] Furthermore, this method employs a joint CTC (Connectionist Temporal Classification) / attention training framework, where the ASR loss based on multi-task learning is represented as L joint Combined with L joint and L d The final loss formula is:
[0056]
[0057] Where α is the weight of the regularization term.
[0058] This implementation demonstrates that the speech generation method, which uses speech fragments constructed from basic phoneme n-grams to generate speech for text in the target domain, possesses the ability to model word linking in the concatenated synthesis method. Furthermore, because the synthesized audio fragments are derived from a large amount of real speech, the synthesized speech exhibits better speaker diversity and reduces the required computational resources. This effectively helps train speech recognition models to achieve domain transfer and avoids overfitting of ASR models on synthesized data.
[0059] This paper provides a detailed experimental demonstration of the proposed method. The ASR model trained on the LIBRISPEECH dataset was adapted to various target domains in the GIGASPEECH target dataset for testing. GIGASPEECH is a multi-domain ASR corpus consisting of 10,000 hours of transcribed speech. In this method, a YouTube section of the GIGASPEECH XL dataset was used. Four different domains with comparable data volumes were selected as target domains. Specifically... Figure 4As shown, a 5-hour development set and a 10-hour test set were separated from the training data for each domain. Only text data from the target domain were used for audio synthesis and model training. The phoneme n-gram dictionary P was constructed using forced alignment results obtained from a TDNN (Time Delayed Neural Network) model in Kaldi.
[0060] The source-domain ASR model was trained on the complete 960-hour LIBRISPEECH dataset. It employed a 12-layer Conformer speech encoder and a 6-layer Transformer decoder with 2048 hidden units. Each layer included eight 64-dimensional self-attention layers. The kernel size of the convolutional modules was 31. For joint CTC attention training, the weights of CTC and attention were empirically set to 0.3 and 0.7, respectively. For most experiments, the weight α of the regularization term was set to 150. An 80-dimensional log-Melbourne filter bank was used, computed every 10ms with a 25ms window length as input to the speech encoder. During adaptation, the model was trained on real speech from the source domain and synthesized speech generated from text in the target domain. The encoder was frozen during adaptation. The Adam optimizer was used with an initial learning rate of 0.001 and 20000 pre-training steps. A joint CTC attention decoding strategy was also employed. During inference, the weights of CTC and attention were set to 0.2 and 0.8, respectively, adjusted to obtain optimal decoding results on the LIBRISPEECH development set. The decoder has 10,000 modeling units for the text sequence. Transformer LMs (language models) are trained on the corresponding text in the target domain. All models are trained until convergence. Experiments are conducted using the ESPnet toolkit.
[0061] This method compares single-speaker and multi-speaker neural TTS (Text-to-Speech) systems. Both systems consist of a FastSpeech2 acoustic encoder and a HiFi-GAN vocoder. The single-speaker and multi-speaker TTS systems were pre-trained using LJSPECH and LIBRITTS, respectively.
[0062] Performance of different system settings, such as Figure 5As shown in the diagram. In the first row, the unadapted model trained on LIBRISPEECH was tested on the four target domains of GIGASPEECH. The last row shows the upper limit performance of the ASR model on the target domain by training it with paired target domain data. Although shallow fusion produces improvements on the target domain by introducing an external LM, better performance can be obtained by tuning the ASR model on synthesized speech-text pairs. Furthermore, performance degradation in the source domain can be suppressed by training on a mixture of real speech from the source domain and synthesized speech from the target domain.
[0063] Comparing the third and fourth rows, the adaptive model based on multi-sensor TTS data consistently produces better results than that based on single-speaker TTS data, indicating that speaker diversity plays a crucial role in pure text domain adaptation based on neural TTS. Although the synthesized speech quality of multi-speaker TTS may be slightly lower than that of single-speaker TTS, the richer speaker diversity in synthesized speech prevents the adaptive model from overfitting to a single speaker.
[0064] Our method's Phoneme SDG (phoneme-level concatenation data generation) was compared with Word SDG (word-level concatenation data generation). In the experiments, Word SDG did not outperform the neural TTS method. This is because the source domain data was reduced from 65,000 hours to 960 hours in LIBRISPEECH, resulting in more OOV (out-of-vocabulary) words and reduced diversity of word-guided speech segments. This drawback is mitigated in our Phoneme SDG method because phoneme n-grams are much richer than words. Compared to Word SDG, our Phoneme SDG method shows consistent improvement in the target domain. Furthermore, most adaptive models using Phoneme SDG produce better results in the source domain than neural TTS methods, showing similar or even better performance in the source domain. We attribute this to the fact that the synthesized speech in SDG consists of speech segments from the source domain, and the SDG method significantly increases the diversity of training data through dynamic data generation.
[0065] The seventh line shows the formula L above. dThe result is achieved by replacing the encoder freezing strategy with a layer-by-layer regularization term. Since the number of trainable parameters increases by unfreezing the encoder, and a regularization term can be used to prevent overfitting, the model adapted using this method achieves the largest relative improvement (i.e., 36.3% / 40.0%) compared to models on the GIGASPEECH development / test set. This is attributed to the larger domain bias of text data in the scientific domain. The smallest relative improvement is observed on the entertainment development / test set (i.e., 18.7% / 19.9%), which can be explained by the insufficient text data in the entertainment domain compared to other domains. This method further validates its compatibility with shallow fusion. The results show that LM shallow fusion can further improve performance on the target domain by sacrificing recognition accuracy in the source domain.
[0066] This method also investigates the influence of the weight α of the regularization term in the final loss formula L mentioned above, and the results are as follows: Figure 6 As shown, when α is too small (α = 50), the regularization is too weak to prevent the adaptive model from overfitting the synthesized speech, resulting in a performance degradation across all test sets. Optimal results are obtained by increasing α to 150.
[0067] In summary, this method generates speech from text in the target domain by concatenating speech segments corresponding to basic phoneme n-grams. Compared to neural TTS methods, this method has lower computational cost and produces more diverse synthesized speech. The ASR model utilizing this method can adapt to dynamically generated speech without affecting training speed. Furthermore, a hierarchical regularization term is introduced to prevent overfitting to the synthesized speech. The effectiveness of the proposed method is validated by applying a model trained on LIBRISPEECH to four different domains in the YouTube section of the GIGASPEECH XL subset. Results show that the relative WER is reduced by approximately 15.0% to 30.0% on the test set from the target domain, while there is almost no deterioration on the test set from the source domain. Performance on the target domain can be further improved by combining the proposed method with shallow LM fusion.
[0068] like Figure 7 The diagram shown is a schematic diagram of a phoneme-based speech domain migration system provided in an embodiment of the present invention. The system can execute the phoneme-based speech domain migration method described in any of the above embodiments and is configured in a terminal.
[0069] This embodiment provides a phoneme-based speech domain migration system 10, which includes: a phoneme sequence determination program module 11, an N-gram sequence conversion program module 12, and a domain migration program module 13.
[0070] The phoneme sequence determination module 11 is used to convert the target domain text from characters to phonemes to obtain a target domain phoneme sequence; the N-gram sequence conversion module 12 is used to convert the target domain phoneme sequence into multiple N-gram phoneme sequences in the target domain according to the N-gram phoneme dictionary, wherein the N-gram phoneme dictionary is constructed from real speech in the source domain; the domain migration module 13 is used to generate speech segments in the target domain with speaker diversity using the multiple N-gram phoneme sequences in the target domain, and generate synthetic audio in the target domain based on the speech segments in the target domain.
[0071] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the phoneme-based speech domain transfer method in any of the above method embodiments.
[0072] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:
[0073] The target domain text is converted from character to phoneme to obtain the target domain phoneme sequence;
[0074] The target domain phoneme sequence is transformed into multiple phoneme N-gram sequences in the target domain according to the phoneme N-gram dictionary, wherein the phoneme N-gram dictionary is constructed from real speech in the source domain;
[0075] The target domain speech segments with speaker diversity are generated using multiple phoneme N-gram sequences in the target domain, and the target domain synthesized audio is generated based on the target domain speech segments.
[0076] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the phoneme-based speech domain transfer method in any of the above method embodiments is executed.
[0077] Figure 8 This is a schematic diagram of the hardware structure of an electronic device for a phoneme-based speech domain transfer method provided in another embodiment of this application, as shown below. Figure 8 As shown, the device includes:
[0078] One or more processors 810 and memory 820, Figure 8 Taking a processor 810 as an example, the device for the phoneme-based speech domain transfer method may also include an input device 830 and an output device 840.
[0079] The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.
[0080] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the phoneme-based speech domain migration method in the embodiments of this application. The processor 810 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the phoneme-based speech domain migration method described in the above embodiments.
[0081] The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 820 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0082] Input device 830 can receive input numerical or character information. Output device 840 may include display devices such as a display screen.
[0083] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, they execute the phoneme-based speech domain migration method in any of the above method embodiments.
[0084] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.
[0085] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0086] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the phoneme-based speech domain migration method of any embodiment of this invention.
[0087] The electronic devices described in this application exist in various forms, including but not limited to:
[0088] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.
[0089] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.
[0090] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.
[0091] (4) Other electronic devices with data processing functions.
[0092] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A phoneme-based method for generating target domain speech from source domain speech, comprising: The target domain text is converted from character to phoneme to obtain the target domain phoneme sequence; The target domain phoneme sequence is transformed into multiple phoneme N-gram sequences in the target domain according to the phoneme N-gram dictionary, wherein the phoneme N-gram dictionary is constructed from real speech in the source domain; The target domain speech segments with speaker diversity are generated using multiple phoneme N-gram sequences in the target domain, and the target domain synthesized audio is generated based on the target domain speech segments.
2. The method according to claim 1, wherein, The step of converting the target domain phoneme sequence into multiple target domain phoneme N-gram sequences based on the phoneme N-gram dictionary further includes: Silence sequences are added to both ends of each phoneme N-gram sequence to simulate pauses in real speech dialogue.
3. The method according to claim 1, wherein, The phoneme N-gram dictionary is constructed from real speech in the source domain and includes: The phoneme N-gram dictionary is generated based on the forced alignment results of real speech in the source domain.
4. The method according to claim 1, wherein, After generating synthetic audio in the target domain, the method further includes: training an automatic speech recognition model using the synthetic audio in the target domain and real speech in the source domain; Specifically, when training with real speech from the source domain, the automatic speech recognition loss of the automatic speech recognition model is subjected to distance regularization.
5. A phoneme-based system for generating target domain speech from source domain speech, comprising: The phoneme sequence determination module is used to convert character-to-phoneme text in the target domain to obtain the target domain phoneme sequence. The N-gram sequence conversion program module is used to convert the target domain phoneme sequence into multiple phoneme N-gram sequences in the target domain according to the phoneme N-gram dictionary, wherein the phoneme N-gram dictionary is constructed from real speech in the source domain; The domain transfer procedure module is used to generate speaker-diverse speech segments in the target domain using multiple phoneme N-gram sequences in the target domain, and to generate synthesized audio in the target domain based on the speech segments in the target domain.
6. The system according to claim 5, wherein, The N-gram sequence conversion module is used for: Silence sequences are added to both ends of each phoneme N-gram sequence to simulate pauses in real speech dialogue.
7. The system according to claim 5, wherein, The N-gram sequence conversion module is used for: The phoneme N-gram dictionary is generated based on the forced alignment results of real speech in the source domain.
8. The system according to claim 5, wherein, The system also includes a model training module for: training an automatic speech recognition model using synthesized audio from the target domain and real speech from the source domain; wherein, when training with real speech from the source domain, the automatic speech recognition loss of the automatic speech recognition model is subjected to distance regularization.
9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-4.
10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-4.