Speech synthesis with foreign segments

Abstract

A method for synthesizing speech based on a textual input includes receiving a textual input including native words of a native language and foreign words of a foreign language and processing the textual input to determine a phonetic representation of the textual input. The processing includes determining a native phonetic symbol representation of the native word and determining a localized phonetic symbol representation of the foreign word. Determining the localized phonetic representation includes forming a foreign phonetic representation of the foreign word using the set of foreign phonemes, and mapping the foreign phonetic representation to the localized phonetic representation according to a model of pronunciation of the foreign word by the local speaker.

Description

Technical Field

[0001] The present invention relates to speech synthesis using foreign segments. Background Art

[0002] Text-to-speech (TTS) systems synthesize speech waveforms based on text input. Some conventional TTS systems process the text input to determine phonetic transcriptions of words or subsets of words (i.e., letters or groups of letters called "graphemes") in the text input. The phonetic transcriptions are provided to a synthesizer, which converts the phonetic transcriptions into speech waveforms that can be output (e.g., using a speaker).

[0003] Some text inputs to a TTS system include fragments of one or more words in a second language. An example of such a text input is "Hello, Mr. Harry Potter" ("J'ai lu HarryPotterhier"), where the TTS system is configured to synthesize native French text as spoken by a French speaker, but Harry Potter is a foreign language text because it is the proper name of a fictional British wizard. One approach may be to ignore the change in language and use the same synthesis rules for the entire input. Another approach may be to switch between French and English synthesis rules (i.e., as if switching between mid-sentences of a native French and native English speaker). However, this approach may not produce synthesized speech that sounds natural. Summary of the invention

[0004] In general terms, methods of speech synthesis accept text having a foreign language segment and produce a synthesized waveform that pronounces the native language text as if it were spoken by a native speaker and pronounces the foreign language segment in the manner in which the native speaker would pronounce it, which may not correspond to the "correct" pronunciation of the foreign language segment by a speaker of the foreign language.

[0005] For example, some TTS systems may recognize that a text input includes both French and English words, and then use French phonemes to synthesize the French words, and use English (e.g., British English) phonemes to synthesize the English words using pronunciation rules inherent to each language. A disadvantage of this technique is that the synthesized speech may sound unnatural because in the middle of a sentence, the French speaker may suddenly appear to switch to speaking perfect English and then switch back to speaking perfect French. Other techniques may replace the English phonemes of the text input with their closest French equivalents, or may use French grapheme-to-phoneme rules to synthesize English segments. The resulting synthesized speech may sound unnatural because certain aspects of the text may be unnaturally omitted from the English portion of the synthesized speech.

[0006] Aspects described herein are by synthesizing the foreign language text embedded in the native language text input in the mode of imitating the native speaker pronunciation foreign language text to solve the above-mentioned shortcomings of traditional technology.For example, many native French speakers are proficient in English at least to a certain extent.Even if French does not include the h phoneme, those French speakers can also recognize the pronunciation of "H" in the name "Harry" in English, and try (may be imperfectly) to pronounce the sound of "H" when saying "Harry".In fact, those native French speakers are using the French phoneme set of expansion to pronounce English words.Aspects described herein will imitate the attempt of "H" in French speaker pronunciation "Harry" by using the expanded French phoneme set that comprises the extra phoneme that French speakers use when pronouncing English words, thereby produce the synthetic speech that sounds more natural and real.

[0007] In a general aspect, a method for synthesizing speech based on text input includes receiving the text input, and processing the text input to determine a phonetic representation of the text input, the text input including native words of a native language and foreign words of a foreign language. The processing includes determining a native phonetic representation of the native words, and determining a nativized phonetic representation of the foreign words. Determining the nativized phonetic representation includes forming a foreign phonetic representation of the foreign word using a set of foreign phonemes, and mapping the foreign phonetic representation to the nativized phonetic representation based on a model of a native speaker's pronunciation of the foreign word.

[0008] These aspects may include one or more of the following features.

[0009] The native phonetic representation may include phonemes from a native phoneme set. The localized phonetic representation may include phonemes from a native phoneme set. The localized phonetic representation may include phonemes from a second phoneme set different from the native phoneme set. The second phoneme set may include a phoneme set of a foreign language.

[0010] The mapping of the foreign phonetic symbol representation to the localized phonetic symbol representation may use context information associated with the foreign phonetic symbol representation. The context information may include a grapheme representation of the foreign text associated with the foreign phonetic symbol representation. The context information may include an alignment of graphemes in the grapheme representation of the foreign text with phonemes in the foreign phonetic symbol representation. The context information may include position information of the phonemes in the foreign phonetic symbol representation.

[0011] The model of the pronunciation of foreign words by native speakers can be based on training data, which includes foreign text phrases and phonetic transcriptions of the pronunciation of the foreign text phrases by native speakers. At least some of the native speakers' pronunciations of foreign text phrases may be mispronunciations. The method can include providing a waveform synthesizer with a combination of native phonetic representations of native words and localized phonetic representations of foreign words for synthesizing speech waveforms. The method can include using a waveform synthesizer to synthesize speech waveforms based on a combination of native phonetic representations of native words and localized phonetic representations of foreign words.

[0012] The method may include configuring a neural network according to a model of native speaker pronunciation of foreign words. Mapping the foreign phonetic representation to the localized phonetic representation according to the model of native speaker pronunciation of foreign words may include applying one or more mapping rules. The method may include identifying foreign words in text input.

[0013] In another general aspect, a system for synthesizing speech based on text input includes an input for receiving the text input, the text input including native words of a native language and foreign words of a foreign language, and one or more processors configured to process the text input to determine a phonetic representation of the text input. The processing includes determining a native phonetic representation of the native words, and determining a localized phonetic representation of the foreign words. Determining the localized phonetic representation includes forming a foreign phonetic representation of the foreign word using a set of foreign phonemes, and mapping the foreign phonetic representation to the localized phonetic representation based on a model of a native speaker's pronunciation of the foreign word.

[0014] In another general aspect, software stored in a non-transitory form on a computer-readable medium includes instructions for causing a computing system to synthesize speech based on text input, the instructions including receiving text input, and processing the text input to determine a phonetic representation of the text input, the text input including native words of a native language and foreign words of a foreign language. The processing includes determining native phonetic representations of the native words, and determining localized phonetic representations of the foreign words. Determining the localized phonetic representation includes forming a foreign phonetic representation of the foreign word using a set of foreign phonemes, and mapping the foreign phonetic representation to the localized phonetic representation based on a model of a native speaker's pronunciation of the foreign word.

[0015] In another general aspect, a method for determining configuration data for configuring a module for mapping foreign phonetic symbol representations to localized phonetic symbol representations includes receiving training data, and processing the training data to form configuration data, the training data including: multiple text representations of foreign language words or phrases and corresponding multiple reference phonetic symbol representations of the foreign language words or phrases.

[0016] These aspects may include one or more of the following features.

[0017] Processing the training data may include: for each textual representation of a foreign word or phrase, forming a foreign phonetic representation of the foreign word or phrase using a set of foreign phonemes, mapping the foreign phonetic representation to a localized phonetic representation based on a model of the foreign word pronunciation by a native speaker, determining a difference between the localized phonetic representation and a reference phonetic representation corresponding to the foreign word or phrase, and updating the model of the foreign word pronunciation by a native speaker based at least in part on the determined difference.

[0018] An advantage of training a mapping from correct foreign pronunciation to target language pronunciation, as opposed to, for example, training a complete grapheme-to-phoneme mapping that maps foreign text to a target phoneme set, is that relatively little training data is needed to represent the way a native speaker would "localize" a foreign segment. Another advantage when using graphemes as part of a method to localize a foreign segment is that because properties of foreign language pronunciation, such as vowel coloring (e.g., very short vowel phonemes, such as the central vowel (schwa) $) may be lost in an English phonetic transcription, these properties can be restored by using graphemes as part of the localization of a foreign segment.

[0019] Another advantage is that different mappings can be trained for different target users. For example, speakers living in a generally bilingual region may pronounce a foreign segment closer to its correct foreign form than speakers from regions where the foreign language is not spoken. In order to provide users from these different regions with a "natural" synthesis of the foreign segment, different mappings may be used.

[0020] Other features and advantages of the invention will be apparent from the following description, and from the claims.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a text-to-speech synthesis system.

[0023] Figure 2 is a detailed example of the operation of a text-to-speech synthesis system.

[0024] Figure 3 Is the phoneme mapping module.

[0025] Figure 4 Is the coverage mapping module.

[0026] Figure 5 It is a system for training phoneme mapping modules.

[0027] Figure 6 It is a phoneme definition table. DETAILED DESCRIPTION

[0028] 1 Overview

[0029] refer to Figure 1 , a text-to-speech synthesis system 100 is configured to receive a text input 102, and process the text input to generate a synthesized speech waveform 104 for presentation using a speaker 106. The text input to the system 100 includes both native language words and foreign language words. Very generally, the system 100 is configured to generate a synthesized speech waveform based on such text input, and the synthesized speech waveform pronounces the native language words in the text input like a native speaker, and imitates how a native speaker would pronounce (and possibly mispronounce) the foreign language words in the text input. As a result, the synthesized speech waveform 104 sounds more natural to a native listener. For simplicity, the rest of this document uses French as a native language and English as a foreign language. However, note that the examples described are not limited to any two languages.

[0030] In some examples, the system 100 includes a language analyzer 108, a French language processing pipeline 110, an English language processing pipeline 112, and a waveform generator 114. The language analyzer identifies French words or phrases 114 in the text input 102, and identifies English words or phrases 116 in the text input 102. The French words or phrases 114 are provided to and processed by the French language processing pipeline 110, and the English words or phrases 116 are provided to and processed by the English language processing pipeline 116. The outputs of the French language processing pipeline 112 and the English language processing pipeline 112 are combined and provided to the waveform generator 114, which processes the combined outputs to generate the synthesized speech waveform 104.

[0031] The French language processing pipeline 110 includes a French grapheme-to-phoneme (G2P) module 118 that recognizes graphemes (i.e., letters or groups of letters) in a French word or phrase 114 and determines the French language phonemes associated with the recognized graphemes. The output of the grapheme-to-French phoneme module 118 is a French phoneme sequence 121 that corresponds to the French word or phrase 114.

[0032] The English language processing pipeline 112 processes English words or phrases 116 to ultimately generate "French + "A sequence of phonemes in a phoneme set. Generally, this phoneme set is denoted as "French +” to illustrate a phoneme set that includes French phonemes and possible additional phonemes that a native French speaker would use when trying to pronounce an English word, but may not need to pronounce a French word. In some examples, the additional phonemes are existing English phonemes and / or new phonemes that are unique to a French speaker trying to pronounce an English word. The English language processing pipeline 112 includes a grapheme to English phoneme module 120, an English to French + A phoneme mapping module 123 and an optional coverage mapping module 124 .

[0033] The grapheme to English phoneme module 120 recognizes graphemes in the English word or phrase 116 and determines the English language phonemes associated with the recognized graphemes. The output of the grapheme to English phoneme module 120 is an English phoneme sequence 122 that corresponds to the English word or phrase 116.

[0034] English phoneme sequences 122 (and optionally graphemes associated with English words or phrases 116 and aligned with the English phonemes) are provided to the English to French + Phoneme mapping module 123, which maps English phoneme sequence 122 to French + Phoneme sequence 126 (as described in more detail below). In some examples, English to French + The phoneme module 123 is implemented as a neural network parameterized by mapping parameters θ 125. Optionally, the overlay mapping module 124 processes French + Phoneme sequences to override certain phoneme mappings using override rules (as described in more detail below).

[0035] French Phoneme Sequence 121 and French + The phoneme sequence 126 is provided to a combiner 128 which combines the two phoneme sequences according to the order of words and phrases in the text input 102 to generate a combined French and French. + Phonemes 130. Combined French and French + The phonemes 130 are provided to the waveform generator 114 , which processes the sequence of phonemes to generate the synthesized speech waveform 104 .

[0036] 2 Examples

[0037] refer to Figure 2 , system 100 receives text input 102 “J'ai luHarryPotterhier” and processes the text input using language analyzer 108 to recognize the words “J'ai lu” and “hier” as French words or phrases 110 and to recognize the word “HarryPotter” as a fragment of an English word or phrase 116.

[0038] The French words or phrases 110, "J'ai lu" and "hier" are provided to the grapheme to French phoneme module 118 (for simplicity, in Figure 2 Grapheme to French phoneme module 118 processes "J'ai lu" to generate French phoneme sequence 121'Ze_'ly ( Figure 6 Including English, French and French used as reference in this article + Definition of phonemes). The grapheme to French phoneme module 118 processes "hier" to generate the French phoneme sequence i.'jER.

[0039] The English word or phrase 116, "HarryPotter" is provided to a grapheme to English phoneme module 120. The grapheme to English phoneme module 120 processes "HarryPotter" to generate an English phoneme sequence 122 'h@R+I_'pA+.t$R+. The English phoneme sequence 122 (and optionally the associated English graphemes) is provided to an English to French + Phoneme Mapping Module 123. English to French + The phoneme mapping module 123 maps the English phoneme sequence 122 to the French + Phoneme sequence 126h a.'R+I_p O.'tE+R+. Note that French + The sequence 126 of phonemes includes the phonemes h and R+, which do not exist in the native French phoneme set but are borrowed from the English phoneme set.

[0040] When English graphemes are fed into English into French + The phoneme mapping module 123 recognizes certain English graphemes and French + Mapping between phonemes, and inserting French when the corresponding English grapheme appears in the text input + phoneme.

[0041] The combiner 128 combines the French phoneme sequence 121 with the French + Phoneme sequences 126 are combined to form combined French and French + The phonemes 130 are provided to the waveform generator 114 to generate the synthesized speech waveform 104.

[0042] 3. English to French + Mapping

[0043] refer to Figure 3 , in some examples, English to French + The phoneme mapping module 123 receives the English phoneme sequence 122 and, for each English phoneme, determines the corresponding mapped French +Phonemes (e.g., mapping module 123 is a sliding mapper that is repeatedly applied to the phonemes in English phoneme sequence 122). Module 123 includes a phoneme mapper 232 (e.g., a neural network) configured using mapping parameters θ 125. Phoneme mapper 123 receives a single English phoneme 234 ( Figure 3 $) in the , and maps English phoneme 234 to the (possibly null) French + phoneme( Figure 3 Note that mapping module 123 does not necessarily generate a single French phoneme based on a single English phoneme. + Rather than using English phonemes, the context of the English phonemes and their corresponding graphemes can be considered. For example, the mapping module 123 can apply a sliding window to the English phoneme sequence 122 and the graphemes corresponding to the English phoneme sequence. The information in the sliding window provides the context that the mapping module 123 uses to determine one or more French + Mapping of phonemes.

[0044] In other examples, English to French + The phoneme mapping module 123 implements sequence-to-sequence mapping, where English phoneme sequences are mapped to French + Phoneme sequences are mapped in a way that takes into account the context of each phoneme (and possibly grapheme) in the larger phoneme sequence.

[0045] As mentioned above, when English graphemes are provided to English to French + When the phoneme mapping module 123 is used, the phoneme mapper 232 is configured to identify the French + For example, the phoneme mapper 232 may use the "e" grapheme 239 to determine that the English phoneme 234 is mapped to E+French + Phonemes 235.

[0046] 4 Coverage Mapping

[0047] refer to Figure 4 In some examples, coverage mapping module 124 implements rules or other heuristics to identify French + The phonemes in the phoneme sequence 126 (or the lack of phonemes) should be replaced by other different phonemes. Figure 4 Chinese, French + The phoneme sequence 126 has a Null phoneme 436 mapped to an "h" English grapheme 437. Rule 438 is applied to identify the mapping of the Null phoneme to the "h" grapheme and to map the Null phoneme to an "h" French grapheme with h. + Phones replace Null phonemes. This way, in some cases, rules can be used to cover English to French + Output of the phoneme mapping module 123.

[0048] exist Figure 4 , decision point 440 determines whether to use the output of rule 438 or the output from French + The mapped phonemes of the phoneme sequence 126. For example, if the output of rule 438 is Null, decision point 440 uses the French + The mapped phoneme 436 of the phoneme sequence 126. Otherwise, the decision point 440 uses the output of the rule 438 and overwrites the mapped phoneme 436.

[0049] 5. Training

[0050] refer to Figure 5 , the training system 500 receives a set of textual English phrases 542 and a corresponding set of transcribed French + A reference phrase 544 of a phoneme sequence is taken as input and processed to determine mapping parameters θ 125 .

[0051] In operation, the training system 500 processes the set of textual English phrases 542 one at a time by reading textual English phrases 546 from the set of textual English phrases 542. The textual English phrases 546 are processed by the grapheme to English phoneme module 520 to generate an English phoneme sequence 522 corresponding to the textual English phrases 546.

[0052] English phoneme sequence 522 (and optionally associated English graphemes) is provided for English to French + Phoneme Mapping Module 523. English to French + The phoneme mapping module 523 maps the English phoneme sequence 522 to the French phoneme sequence 524 based on the current mapping parameter θ125. + Phoneme mapping sequence 526.

[0053] French associated with text English phrase 546 + The phoneme mapping sequence 526 and the French phoneme reference sequence 548 are provided to the loss calculation module 550, which calculates the loss value L θ , which represents French + Phoneme Mapping Sequence 526 and French + The difference between the phoneme reference sequence 548. The loss value L θ is provided to the optimization module 552, which updates the mapping parameters θ125 to minimize the loss value of the set of training data.

[0054] The training system 500 repeats the parameter updating process for all English phrases 542 so that the mapping parameters θ converge.

[0055] In some examples, French + Phoneme reference sequences are aligned (automatically or manually) to textual English phrases (or corresponding English phoneme sequences) to improve training performance.

[0056] In some examples, each French + The phoneme reference sequences are manually transcribed. In other examples, each French + The phoneme reference sequence is obtained by using the French + In such an example, the mapping parameters θ would configure the mapper to simulate the population of French speakers used to generate the training data (i.e., a population of French speakers who are fluent in English would result in different mapping behavior than a population of French speakers who are only familiar with English).

[0057] In other examples, each French + The phoneme reference sequence is obtained by using the French phonemes produced by a French speaker reading an English phrase. + A phoneme set is generated to perform phoneme recognition.

[0058] 6 Alternatives

[0059] The above example uses a neural network and optional rules to map English phonemes to an expanded set of French phonemes (French + ). However, in some examples, instead of using a neural network, a rule-based approach is sufficient. Alternatively, other machine learning methods (such as using a decision tree) can be used instead of a neural network.

[0060] The above example maps English phonemes to phonemes in an augmented French phoneme set. However, there is no requirement that the French phoneme set be augmented—the native French phoneme set may include enough phonemes to train a mapper to generate speech that mimics a native French speaker trying to pronounce an English word.

[0061] While the above-described systems produce waveforms that are presented as acoustic signals, the output may include representations from which waveforms may be generated. For example, a sequence of phonemes may be processed to form a spectrogram (energy versus frequency versus time) from which a waveform may be determined in a language-independent manner.

[0062] Alternative mapping methods can be used to process foreign phoneme sequences to form target phoneme sequences. For example, a sequence-to-sequence neural network (e.g., a transformer neural network or a recursive neural network such as a Long Short Term Memory (LSTM) network) can be used. An alternative training method can be used to determine the target phoneme sequence by phoneme recognition performed by one or a combination of native speakers who utter the foreign segments, manually transcribing the utterances of these native speakers. In addition, the mapping can be trained to directly match the audio of the native speaker.

[0063] In another approach, different foreign segments can be treated differently, for example based on their length. For example, it may be that short segments are mapped to be close to the native language, while longer segments retain more of their correct foreign pronunciation. Similarly, named entities may retain their correct pronunciation better than ordinary expressions. Therefore, optionally, there can be a mapping process that can change the degree to which foreign segments are mapped to native pronunciation. In addition, the user can prefer the degree of "nativeness" of the pronunciation, which can be set by the user.

[0064] In some examples, where neural networks or rule-based mappings cannot correctly capture anomalies, a lexicon (lexica) is used to cover English phonemes to an extended French phoneme set (French + ) at least some portion of the mapping.

[0065] As described above, in some examples, graphemes and phonemes are aligned prior to processing using a classification model or a rule-based mapper. Different types of grapheme-phoneme alignment techniques may be used, such as expectation-maximization-based alignment, phonetic similarity-based alignment, constraint-based alignment, alignment using an integer programming framework, alignment by aggregation (i.e., a combination of integer programming and expectation-maximization), or any other suitable alignment technique.

[0066] 7 Implementation

[0067] For example, the above method can be implemented using a programmable computing system that executes appropriate software instructions, or can be implemented in appropriate hardware (such as a field-programmable gate array (FPGA) or the like) or in some hybrid form. For example, in a programming method, the software can include processes in one or more computer programs executed on one or more programming or programmable computing systems (which can have various architectures such as distributed, client / server or grid), each computing system including at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), at least one user interface (for receiving input using at least one input device or port, and providing output using at least one output device or port). The software can include one or more modules of a larger program, for example, providing services related to the design, configuration and execution of a data flow graph. The modules of the program (e.g., elements of a data flow graph) can be implemented as data structures or other organized data that conform to a data model stored in a data repository.

[0068] The software may be stored in a non-transitory form for a period of time (e.g., the time between refresh cycles of a dynamic memory device such as a dynamic random access memory (RAM)) using the physical properties of the medium (e.g., surface pits and lands, magnetic domains, or charge), such as contained in a volatile or non-volatile storage medium or any other non-transitory medium. In preparation for loading instructions, the software may be provided on a tangible, non-transitory medium, such as a compact disc-read only memory (CD-ROM) or other computer-readable medium (e.g., readable by a general or special computing system or device), or the software may be transmitted (e.g., encoded in a propagating signal) to a computing system that executes the software over a communication medium of a network. Some or all of the processing may be performed on a dedicated computer, or using dedicated hardware, such as a coprocessor or field-programmable gate array (FPGA) or a dedicated application-specific integrated circuit (ASIC). The processing may be implemented in a distributed manner, with different parts of the calculations specified by the software being performed by different computing elements. Each such computer program is preferably stored or downloaded onto a computer-readable storage medium (e.g., solid-state memory or medium, or magnetic or optical medium) of a storage device accessible to a general or special purpose programmable computer for configuring and operating the computer when the storage device medium is read by the computer to perform the processes described herein. The system of the present invention may also be considered to be implemented as a tangible, non-transitory medium configured with a computer program, wherein the medium so configured causes the computer to operate in a specific and predefined manner to perform one or more of the process steps described herein.

[0069] Some embodiments of the present invention have been described. However, it should be understood that the foregoing description is intended to illustrate rather than limit the scope of the present invention, which is defined by the scope of the claims. Therefore, other embodiments are also within the scope of the claims. For example, various modifications may be made without departing from the scope of the present invention. In addition, some of the steps described above may be sequence-independent and therefore may be performed in an order different from that described.

Claims

1. A method for synthesizing speech based on text input, the method comprising: receiving the text input, wherein the text input includes native words in a native language and loanwords in a foreign language; as well as Processing the text input to determine a phonetic representation of the text input, the processing comprising: determining a native phonetic representation of the native word, and Determining the localized phonetic symbol representation of the foreign word, wherein determining the localized phonetic symbol representation comprises: Using the foreign phoneme set, forming a foreign phonetic representation of the foreign word, and The foreign phonetic symbol representation is mapped to the localized phonetic symbol representation according to a model of a native speaker's pronunciation of a foreign word.

2. The method according to claim 1, wherein: The native phonetic representation includes phonemes from a native phoneme set.

3. The method according to any one of claims 1 to 2, wherein: The localized phonetic representation includes phonemes from a local phoneme set.

4. The method according to claim 3, wherein: The localized phonetic representation also includes phonemes from a second phoneme set that is different from the native phoneme set.

5. The method according to claim 4, wherein: The second phoneme set includes a phoneme set of the foreign language.

6. The method according to any one of claims 1 to 5, wherein: The mapping of the foreign phonetic symbol representation to the localized phonetic symbol representation uses context information, the context information being associated with the foreign phonetic symbol representation.

7. The method according to claim 6, wherein: The context information includes a grapheme representation of the foreign text, and the grapheme representation of the foreign text is associated with the foreign phonetic symbol representation.

8. The method according to claim 7, wherein: The context information also includes alignments of graphemes in the grapheme representation of the foreign text and phonemes in the foreign phonetic symbol representation.

9. The method according to any one of claims 6 to 8, wherein: The context information includes position information of phonemes in the foreign phonetic symbol representation.

10. The method according to any one of claims 1 to 9, wherein: The model of a native speaker's pronunciation of a foreign word is based on training data including foreign text phrases and phonetic transcriptions of the native speaker's pronunciation of the foreign text phrases.

11. The method according to claim 10, wherein: At least some of the native speaker's pronunciations of foreign text phrases are mispronunciations.

12. The method according to any one of claims 1 to 11, further comprising: A combination of the native phonetic representation of the native word and the localized phonetic representation of the foreign word is provided to a waveform synthesizer for use in synthesizing a speech waveform.

13. The method according to claim 12, further comprising: The speech waveform is synthesized using the waveform synthesizer based on a combination of the native phonetic representation of the native word and the localized phonetic representation of the foreign word.

14. The method according to any one of claims 1 to 13, further comprising: The neural network is configured according to a model of the native speaker's pronunciation of the foreign word.

15. The method according to any one of claims 1 to 14, wherein: According to the model of the native speaker's pronunciation of the foreign word, mapping the foreign phonetic representation to the localized phonetic representation includes: applying one or more mapping rules.

16. The method of any one of claims 1 to 15, further comprising identifying the foreign words in the text input.

17. A system for synthesizing speech based on text input, the system comprising: Input, for receiving the text input, the text input includes native words of the mother tongue and foreign words of the foreign language; as well as One or more processors configured to process the text input to determine a phonetic representation of the text input, the processing comprising: determining a native phonetic representation of the native word, and Determining the localized phonetic symbol representation of the foreign word, wherein determining the localized phonetic symbol representation comprises: Using the foreign phoneme set, forming a foreign phonetic representation of the foreign word, and The foreign phonetic symbol representation is mapped to the localized phonetic symbol representation according to a model of a native speaker's pronunciation of a foreign word.

18. Software stored in non-transitory form on a computer readable medium, the software comprising instructions for causing a computing system to synthesize speech based on text input, the instructions comprising: receiving the text input, wherein the text input includes native words in a native language and loanwords in a foreign language; as well as Processing the text input to determine a phonetic representation of the text input, the processing comprising: determining a native phonetic representation of the native word, and Determining the localized phonetic symbol representation of the foreign word, wherein determining the localized phonetic symbol representation comprises: Using the foreign phoneme set, forming a foreign phonetic representation of the foreign word, and The foreign phonetic representation is mapped to a localized phonetic representation according to a model of a native speaker's pronunciation of the foreign word.

19. A method for determining configuration data, the configuration data being used to configure a module for mapping foreign phonetic symbol representations to localized phonetic symbol representations, the method comprising: receiving training data, the training data comprising: a plurality of text representations of foreign language words or phrases and a corresponding plurality of reference phonetic representations of the foreign language words or phrases; and The training data is processed to form the configuration data.

20. The method according to claim 19, wherein: Processing the training data includes: For each textual representation of a foreign language word or phrase: Use foreign phoneme sets to form foreign phonetic representations of foreign words or phrases. According to a model of the pronunciation of foreign words by native speakers, the foreign phonetic representation is mapped to the localized phonetic representation, determining a difference between the localized phonetic representation and a reference phonetic representation corresponding to the foreign language word or phrase, and The model of the native speaker's pronunciation of the foreign word is updated based at least in part on the determined differences.

Images