Transliteration for speech recognition training and scoring

Abstract

A method, system, and apparatus for transliteration for speech recognition training and scoring, including a computer program stored on a computer-readable storage medium. In some embodiments, language examples are accessed, some of which include words in a first script and words in one or more other scripts. At least a portion of some of the language examples are transliterated into the first script to generate a training data set. A language model is generated based on the occurrence of different sequences of words in the training data set in the first script. The language model is used to perform speech recognition for an utterance.

Description

Background Art

[0001] Many examples of spoken and written language include words from two or more different languages ​​or dialects. Many speech recognition systems have difficulty accurately transcribing speech that combines words from different languages ​​or dialects. Similarly, it is often difficult to train a speech recognition system using examples of written language that include words in different languages ​​or dialects, especially languages ​​from multiple writing systems. Summary of the invention

[0002] In some embodiments, language examples can be processed to transliterate at least some words into different scripts to enhance the training of the speech recognition model and provide a more accurate evaluation of the speech recognition model. In some cases, written language examples, such as written language examples used to train or test a language model, can include words written in one script (e.g., a first script or a primary script) and one or more words written in different scripts. Before training or evaluating the model, the language sample can be standardized by transliterating words outside the script into the primary script. This can be done for all words outside the script, or it can be done more selectively (e.g., not transliterating proper names). The generated model provides better accuracy than models that exclude examples with mixed scripts from training and models that use mixed scripts without transliteration.

[0003] Often, using multiple scripts in a single phrase or sentence can make it difficult for a language model to properly learn the correct usage of words that are not in the primary script. As a result, examples with mixed scripts are often removed from the training dataset so that they are not used in language model training. This has the disadvantage of limiting the number of examples the model can learn from, and in particular, eliminates the opportunity for the model to learn context and usage where words from multiple scripts are used together. To improve recognition accuracy, transliteration can be used to standardize the training dataset to the desired script, thereby improving modeling accuracy.

[0004] Transliteration of language model training data reduces inconsistencies in transcriptions, provides better standardization, and improves the overall performance of automatic speech recognizer systems. This feature allows for augmentation of language model training data and enforces the recognizer's assumptions to conform to one writing system. In contrast, if model training is limited to allowing all the speech of the secondary writing system under training, the resulting model will tend to output more speech in the secondary writing system, further increasing the already existing writing script mismatch between the model's assumptions and the transcription truth. Even if two words refer to the same semantics and pronunciation, it will have word counts scattered across two or more representations - for example, a word in the first script and a word in the second script. By transliterating the training data, the model output remains primarily in the desired script, while improving accuracy as word counts for identical words are merged for instances in any script.

[0005] In some implementations, a method performed by one or more computers includes: accessing a data set indicating language examples for a first script, wherein at least some of the language examples include words in the first script and words in one or more other scripts; transliterating at least a portion of some of the language examples to the first script to generate a training data set having words transliterated to the first script; and generating a speech recognition model based on occurrences of sequences of words in the training data set having words transliterated to the first script. The method may optionally further include performing speech recognition on an utterance using the speech recognition model.

[0006] In some implementations, the speech recognition model is a language model, an acoustic model, a sequence-to-sequence model, or an end-to-end model.

[0007] In some implementations, transliteration includes mapping different glyphs representing text from different scripts to a single standardized transliterated representation.

[0008] In some implementations, transliterating the language example includes transliterating words in the language example that are not in the first script into the first script.

[0009] In some implementations, transliterating the language example includes: accessing a blacklist of terms in a script different from the first script; and bypassing transliteration of instances of terms from the blacklist that occur in the language example.

[0010] In some implementations, transliterating the language example includes generating an altered language example in which words written in a second script different from the first script are replaced with one or more words in the first script that approximate acoustic characteristics of words in the first script.

[0011] In some implementations, words written in the second script are individually transliterated into the first script on a word-by-word basis.

[0012] In some embodiments, the method includes: determining a test set of language examples for testing the speech recognition model; generating a standardized test set by transliterating words in the language examples in the test set that are not written in the first script into the first script; obtaining the output of the speech recognition model corresponding to the language examples in the test set; standardizing the output of the speech recognition model by transliterating words output by the speech recognition model that are not written in the first script into the first script; and determining the error rate of the speech recognition model based on a comparison of the standardized test set with the output of the standardized speech recognition model.

[0013] In some embodiments, the error rate is a word error rate, and wherein the method includes, based on the word error rate: determining whether to continue training or terminate training of the speech recognition model; changing a training data set used to train the speech recognition model; setting the size, structure or other features of the speech recognition model; or selecting one or more speech recognition models for a speech recognition task.

[0014] In some implementations, the method includes determining a modeled error rate for the speech recognition model, wherein acoustically similar words written in any of a plurality of scripts are accepted as correct transcriptions, while not penalizing output of words in a script that differs from a corresponding word in a reference transcription.

[0015] In some implementations, the method includes determining a rendering error rate for the speech recognition model, the rendering error rate being a measure of the difference between the scripts of words in the output of the speech recognition model relative to the scripts of corresponding words in a reference transcription.

[0016] In some implementations, the transliteration is performed using a finite state transducer network that is trained to perform the transliteration into the first script.

[0017] In some implementations, transliterating includes, for at least one language example, performing multiple rounds of transliteration between scripts to arrive at a transliterated representation in the first script included in a training data set in the first script.

[0018] In some embodiments, the method includes determining a score indicating the degree of mixing of scripts in the language sample; based on the score, selecting parameters for tailoring a finite state transducer network for transliteration; selecting parameters for tailoring the speech recognition model; or selecting a size or structure for the speech recognition model.

[0019] In some embodiments, generating the speech recognition model includes: after transliterating at least a portion of some language examples into the first script, determining, by the one or more computers, the number of occurrences of different word sequences in a training data set in the first script; and generating the speech recognition model based on the number of occurrences of different word sequences in the training data set in the first script by the one or more computers.

[0020] In some implementations, the speech recognition model includes a recursive neural network, and generating the speech recognition model includes training the recursive neural network.

[0021] The present disclosure also provides a method for performing speech recognition, comprising: receiving audio data representing an utterance by one or more computers; and mapping the audio data to text (or some other symbolic representation) representing the utterance using a speech generation model by one or more computers, wherein the speech recognition model is generated in advance according to any content disclosed herein. It should be understood that the computer used to generate the speech recognition model may be different from the computer used to perform speech recognition. The method may further include outputting text representing the output.

[0022] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the operations of the method.

[0023] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a block diagram illustrating an example of a system for transliteration of speech recognition and evaluation.

[0025] Figure 2 is a diagram showing an example of a finite state transducer network for transliteration.

[0026] Figure 3 is a graph showing the error rate relative to the amount of transcoding in a data set.

[0027] Like reference numbers and numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0028] Figure 11 is a diagram showing an example of a system 100 for transliteration of speech recognition and evaluation. The system includes a computer system 110, which may include one or more computers located together or remotely from each other. The computer system 110 has a transliteration module 120, a model training module 130, and a scoring module 140. Figure 1 In the example of , system 100 is used to train and evaluate a speech recognition model, such as language model 150 .

[0029] In this example, a set of language examples 112 is obtained from various sources such as query logs, web pages, books, human or machine recognized speech transcriptions, etc. The language examples 112 are mainly in the first script. As used herein, the term "script" generally refers to a writing system. A writing system is a symbol system used to represent a natural language. Examples of scripts that can be used with the technology disclosed herein include Latin, Cyrillic, Greek, Arabic, Indic, or other writing systems. The language model 150 will be trained to provide output that primarily represents text in the first script. However, in many cases, phrases or sentences written primarily in one script often contain one or more words written in another script. The language examples 112 typically include primarily examples written purely in the first script, but also include some language examples that include words in another script and a combination of words in the first script.

[0030] The transliteration module 120 processes the language example 112 to generate a standardized data set of the first script. Before training the language model 150, the transliteration module 120 processes the language example 112 to transliterate instances of words that are not in the first script to the first script. The transliteration module 120 can use a finite state transducer (FST) network to perform transliteration. The relationship between phonemes and words of different scripts can be learned by analyzing the language example 112 or from other data. In some embodiments, each individual word is transliterated separately, for example, on a word-by-word basis to achieve high-quality correspondence in transliteration. The process can optionally consider the context of the surrounding words to provide high-precision transliteration. Through the transliteration process, the generated word transliterated into the first script can be a word with a pronunciation that matches or is similar to the pronunciation of the corresponding original word that appears in the language example 112. Therefore, the transliteration process can be for the original word that is not originally written in the first script, using a replacement word in the first script that represents the same or similar acoustic features or sounds as the original word, to change the writing system.

[0031] In some embodiments, all words that are not in the first script are transliterated to the first script. In other embodiments, transliteration is performed selectively. For example, the transliteration blacklist 126 can indicate words or phrases that should not be transliterated. The transliteration module 120, therefore, retains these terms in their original scripts when generating training data and test data, even if the original script is different from the first script. If the proper names of people, place names, companies and other entities, such as people, place names, companies, and other entities, are included, which may be more common or more recognizable in the local writing system than the transliterated version, the blacklist may be particularly useful. For example, even among texts that are mainly in other scripts such as Indic, Cyrillic, Kanji, Kana, Kanji, etc., some names (such as "George Washington", "New York" or "Google") are preferably retained in the Latin script. Including words outside the script in the training data set can allow the language model 150 to learn to predict the output of these words in its local script, even if these scripts are different from the main script.

[0032] Based on the language examples 112, the transliteration module 120 generates script-normalized training data 122 for training the language model 150. The transliteration module 120 also generates script-normalized test data 124 for testing the language model 150. Testing can occur at various stages, such as after a certain number of tests are completed. The language model 150 can be tested repeatedly between training processes until a desired performance level is achieved.

[0033] In some embodiments, the language model 150 is configured to receive data representing pronunciations or language units representing a language sequence, for example, data indicating the pronunciation of a language sequence, as input. For example, the input may indicate a series of calls (which may be context-dependent or context-independent), or a distribution of scores for a set of calls. These pronunciations may be determined for the language examples 112 in any of a variety of ways. For language examples for which audio data is available, the pronunciations may be the output of an acoustic model of the audio data. For language examples for which there is no corresponding audio data, such as examples from web pages, books, keyboard-entered queries, etc., the system may use a pronunciation generator 126 to automatically generate pronunciations from written text. The pronunciation generator 126 may access a lexicon 128 that indicates the pronunciations of words in the language of the language example 112, for example, a mapping of a grapheme sequence to a phoneme sequence to perform a conversion of graphemes to phonemes. In addition, for some language examples, words may have pronunciations provided by manual annotations by linguists.

[0034] although Figure 1A pronunciation generator 126 and vocabulary 128 are shown, and these elements are optional in some embodiments. For example, a language example 112 that already has a corresponding pronunciation associated with the text can be received. Moreover, in some embodiments, the language model 150 does not receive language unit information as input, but only receives data indicating a grapheme or word sequence. For example, some language models can be used for second-pass re-scoring of candidate transcriptions and therefore can receive data indicating a suggested word sequence. In this case, the language model can receive data indicating the words themselves, rather than receiving language units of speech, and can be configured to provide a score that indicates the likelihood of training the language model for the entire sequence.

[0035] The model training module 130 trains the language model 150. For example, the language model 150 can be a statistical language model, such as an n-gram model, which is generated based on the number of occurrences of different words and phrases in the script normalized training data 122. Other types of models, such as network models, can be used in addition or alternatively. For example, the language model 150 can include a neural network, and the model training module 130 uses back propagation of errors, stochastic gradient descent, or other techniques to train the neural network. The neural network can be a recurrent neural network, such as a neural network including one or more layers of long short-term memory (LSTM) units. The model can be trained by minimizing an objective function, such as a connectionist temporal classification (CTC) objective function, a state-level minimum Bayesian risk (sMBR) objective function, or other objective functions.

[0036] Typically, the language model 150 is trained to predict, for example, the relative likelihood of occurrence of an indicative word based at least in part on the language context, such as one or more surrounding words or phrases. Thus, the language model 150 can be configured to provide an output indicating the probability that different words in the vocabulary will occur in the presence of one or more immediately preceding words. The prediction can also be based on acoustic or language units, such as the output of a speech or language model indicating a phone sequence. When used with an acoustic model indicating the sounds present in the speech, for example, as a phone or other language unit, the language model 150 can indicate which words and word sequences best represent those sounds based on patterns of actual language usage observed from the script normalized training data 122. The language model 150 can be used to generate scores, such as probability scores or confidence scores, indicating the relative likelihood of different words following each other, which can be used to generate a speech lattice, and then a beam search process can be used to determine the best path through the lattice representing the most likely transcription to be considered, such as the path with the highest score or the lowest cost.

[0037] After or during training, the scoring module 140 is used to evaluate the accuracy of the language model 150. The results of the evaluation include the generation of one or more scores 142 indicating the performance of the language model 150. The scoring module 140 can provide examples from the scripted standardized test data 124 as input to the language model 150, so that the language model 150 generates an output, such as a prediction or probability score, for each input example. The output of the language model can be further processed, for example, using lattice search and beam search or by other techniques, to generate a language sequence output for each input example.

[0038] Optionally, words in the output sequence that are not in the first writing system are transliterated using a transliteration module 120 (which is replicated in the figure for clarity). For a language model 150 generated as described above, the model 150 learns to indicate only words in the primary first script, except for words in the blacklist 126. As a result, the model so trained will indicate the output sequence in the first script. However, for a more general approach, and to allow evaluation of all types of language models, including language models that were not trained under the same conditions or datasets, the output sequence can be normalized using transliteration for more accurate comparison.

[0039] Scoring module 140 may generate a word error rate that indicates the rate at which the output sequence includes words that differ from the original language example 112. However, even if two words are phonetically equivalent to each other (e.g., represent the same sound and semantics), a conventional word error rate calculation will consider a word incorrect if it is in a script that differs from the original. When conversions between languages ​​and writing systems are common, this may artificially increase the apparent error rate when the model effectively predicts the correct word. To avoid this and other inaccuracies, scoring module 140 may generate a modified word error rate, referred to as a modeled error rate, that compares the output sequence normalized to the first script to test data 124 normalized to the first script. As a result, even if the original language example has one or more equivalent words written in a different script, no errors are counted when language model 150 generates a language sequence entirely in the first script. A separate score, the script error rate, is generated to measure the rate at which the script of the output sequence differs from the script of the original language example 112.

[0040] The evaluation score 142 from the scoring module 140 can be provided for output to the user and can also be used to manage the training of the language model 150. For example, the modeling error rate can be used to determine whether to continue training the language model 150 or terminate training the language model 150. Training can be set to continue until the error rate is below a threshold level. As another example, based on one or more error rates, the computer system 110 can change the training data set used to train the language model 150, for example, to introduce a different or expanded data set to obtain better accuracy. As another example, based on one or more scores 142, the computer system 110 can set the size, structure or other characteristics of the language model 150. As another example, based on the evaluation of the language model 150 and the potential score for evaluating one or more other language models, the computer system 110 can select one or more language models for performing a speech recognition task.

[0041] In some embodiments, other scores are determined and used to adjust the training of the language model 150 and / or the transliteration module 120. For example, the computer system 110 can obtain data indicating the rate of script mixing, such as "code switching", that occurs for the language in general or in a particular data set (e.g., language examples 112). Using the score, the system can select parameters for tailoring a finite state transducer network for transliteration, can select parameters for tailoring a language model, or select the size or structure of a language model. In fact, scores can be used to set many different parameters for the structure, training, and operation of the language model 150, including the selection of a data development set (e.g., a validation set used to adjust the training algorithm and prevent overfitting), a training data set, a model size, a learning rate during training, the type of model used (e.g., n-gram, neural network, maximum entropy model, etc.), or an output target set for the model (e.g., to predict words, word fragments, or graphemes).

[0042] In general, the techniques in this application can be used to train and test any of a variety of different types of models, not just language models. For example, transliteration data can be used to train and score language models, acoustic models, sequence-to-sequence models, and / or end-to-end models (e.g., models that receive acoustic information or features and provide outputs indicating the likelihood of a word, word fragment, or grapheme). A sequence-to-sequence model can map an input sequence to an output sequence. For example, a sequence-to-sequence model can receive acoustic information or features representing one or more spoken words and produce symbolic outputs (e.g., text) representing those words.

[0043] Code-switching is a common phenomenon in many multilingual communities. Code-switching generally refers to a speaker switching between languages ​​within a single utterance. Conventional word error rate (WER) measures are insufficient to measure the performance of code-mixed languages ​​due to ambiguities in transcription, spelling errors, and borrowing words from two different writing systems. These rendering errors artificially inflate the WER of automatic speech recognition (ASR) systems and complicate evaluation. Furthermore, these errors make it more difficult to accurately evaluate modeling errors originating from code-switched languages ​​and acoustic models. As discussed further below, a new metric, transliteration-optimized word error rate (toWER), can eliminate many of these irregularities by mapping all text to one writing system.

[0044] These techniques can also be used to improve acoustic and language modeling for bilingual code-switched utterances. Example Indian languages ​​are discussed in detail, although the techniques can be used for any combination of languages ​​and writing systems. The transliteration approach can be used to normalize the data for three types of language models, namely, conventional n-gram language models, maximum entropy based language models, and long short-term memory (LSTM) language models along with the state-of-the-art connectionist temporal classification (CTC) acoustic models. The robustness of the proposed approach is demonstrated on several Indian languages ​​in the voice search business, achieving significant improvements in ASR performance, e.g., up to 10% over the state-of-the-art baselines.

[0045] Code switching is common in Hindi-English, Bengali-English, Arabic-English, and Chinese-English, as well as many other bilinguals. When a word from a foreign language (e.g., English) becomes part of the vocabulary of a native language (e.g., Hindi), the distinction between code switching, loanwords, and the creation of new words in the vocabulary of the native language is often not very clear and continuous. This phenomenon makes transcription of code-switched speech difficult and inconsistent, resulting in the same word being transcribed using different writing systems. These inconsistencies can result in incorrect count assignments between words with similar acoustic and lexical contexts in acoustic and language models.

[0046] Several approaches have been proposed for code-switching speech recognition. One approach uses multi-pass speech recognition, where code-switch regions are first identified by a language identification method that uses only acoustic information. Subsequently, a second pass of merging and transcription may be used to recognize the speech segments using the corresponding monolingual acoustic and language models. Such approaches not only require multiple passes, but also rely on the accuracy of code-switch detection and language identification. They also fail to capture contextual information at code-switch boundaries. Other approaches require constraints such as a unified phonetic alphabet across multiple languages, modeling pronunciation variations in the vocabulary, and other techniques that do not consistently improve speech recognition performance.

[0047] The technique in this paper provides a new strategy based on transliteration for improving ASR performance. The weighted finite state transducer (WFST) has been used for speech recognition decoding, where the WFST representing the context-dependent phone sequence model (C), pronunciation vocabulary (L), and language model (G) can be composed into a single large transducer. or CLG for short), which maps context-dependent phonetic sequences to word sequences. In code-switching languages, the speaker mixes the romanized sequences with the local language script, such as Devanagari in the case of Hindi. The processing flow takes a similar approach, using transliteration models in both the acoustic and language models to capture the code switching. A series of WFST optimizations provide improvements to achieve accuracy, latency, and memory usage operating points to operate efficiently. Importantly, the system demonstrates significant improvements in ASR performance for Indian languages ​​using a transliteration-based approach.

[0048] In the following, some challenges in speech recognition for Indian languages ​​are discussed. A new metric, transliteration-optimized word error rate (toWER), is introduced, along with the proposed method and WSFT optimization. Then, taking Hindi as an example, the importance of evaluating ASR systems by eliminating code switching is discussed, and the problem of scoring in a common transliteration space is discussed. Incorporating transliteration into the training of acoustic and language models can significantly improve the performance. Finally, the generalizability of the proposed method is verified with examples from other Indian languages.

[0049] While the techniques in this paper are applicable to code-switching between any two languages, Indian languages ​​present particularly interesting challenges. In India, bilingualism is widespread and code-switching between the native language and English occurs frequently. Table 1 shows the distribution of Latin scripts seen in the two corpora used to build language models for Indian languages. The corpus containing typed search queries contains far more Latin scripts than the corpus containing transcriptions of spoken queries. This appears to be due to the transcription conventions that transcribers try to adhere to, whereas web-based search queries do not have any restrictions.

[0050]

[0051] Table 1

[0052] Distribution of Latin scripts in Indic languages ​​in two different corpora containing written and spoken queries

[0053]

[0054] Table 2

[0055] Table 1, which includes examples of Devanagari and Latin writing systems, shows the wide distribution of code switching.

[0056] A detailed analysis of the proposed approach is discussed below using Hindi, a language with a large amount of code-switching and training data, and the generalization of the approach and its impact on other Indian languages ​​is also shown.

[0057] Code switching exists in multiple writing systems. For example, Hindi uses the Devanagari script, while Urdu uses the Arabic writing system. Since most Hindi speakers are bilingual, code switching is a part of daily life and the phenomenon often occurs in casual conversations, voice search queries, and presentations, leading to what is commonly referred to as Hinglish. This type of code switching can occur in sentences at the phrase level. Table 2 presents some examples of commonly transcribed speech. The first column shows the commonly used mixed writing system. The second column shows the equivalent text in the Latin script to simplify readability and illustrate the mix of Hindi and English seen in the data. Despite instructions to transcribe Hindi in Devanagari script and English origin words in Latin script, bilingual transcribers can observe many inconsistencies in the resulting transcriptions. Like other Indian languages, Hindi is romanized in user-generated text on social and news media, especially with named entity mentions, URLs, numeric entities, and acronyms, thus making the transcription of Hindi words in Devanagari more difficult for transcribers. These types of inconsistencies directly affect the definition of an error and the metric used to evaluate ASR systems, the word error rate (WER). These are called rendering errors.

[0058] The variability in usage of both the native language (Hindi) and the foreign language (English) makes it challenging to model the contexts in which code-switching occurs. While several approaches using linguistic, prosodic, and semantic cues have been proposed to model and capture the code-switching phenomenon, few have been proposed and successfully improved the performance of ASR systems. Lack of consistency and improper normalization in transcriptions also affect the modeling capabilities of the linguistic and acoustic models. This type of error is called modeling error. The techniques discussed next provide a unified approach to address both modeling and rendering errors.

[0059] Transliteration is the process of converting a sequence from one writing system to another. The process of transliterating Indic words into Latin script is particularly challenging due to the large number of combinations of consonants, vowels, and diacritics that result in non-unique mappings. Notably, non-standard spellings exist in both scripts, for example, loanwords have variable possible spellings in Devanagari, while Hindi words have variable romanized forms.

[0060] The universal transliteration approach is applicable to solving code conversion between any two languages ​​or writing systems. Transliteration can be done efficiently via weighted finite state transducers. To generate the source data, human transcribers are asked to transcribe speech in the native writing script (in this case, Devanagari), but certain commonly used English words will be written in Latin script. Therefore, unlike unrestricted text input through a keyboard, the content and scope of the input from the two writing systems is limited to what is said in the utterance. However, due to the lack of a canonical transliteration between the two writing systems and the inconsistencies between the transcriptions, a large number of modeling and rendering errors are introduced. To address this issue, the transliteration transducer T consists of three transducers: where I maps input Unicode symbols to symbols in a paired language model, P is a bigram paired language model that maps between symbols in two writing scripts, English and Devanagari, and O maps paired language model symbols to target output Devanagari symbols (e.g. Figure 2 ). The conditional probability of the transliterated word is obtained by dividing the joint probability of T by the marginalized sum over all input and output sequences. This computation can be done efficiently by computing the shortest path in T.

[0061] To improve the performance of transliteration relative to the memory, speed, and latency requirements for building large-scale language models, several optimizations can be performed.

[0062] In some embodiments, the transliteration transducer computes the shortest path, and significant speed improvements are obtained by effectively pruning the search space. All paths with a score below a pruning threshold are discarded. This threshold is determined empirically so as not to affect ASR performance. A pruning weight threshold of 5 is determined to be a good operating point, especially since the best path is the path of most interest.

[0063] In some embodiments, it is important to use ε-transitions to reduce the number of deletions and insertions when reducing ε-cycles in WFSTs. ε removal was implemented using parallelism and utilized eight threads.

[0064] In some embodiments, the operation for ε removal results in a dramatic increase in memory usage, making the transliteration process unusable for large-scale language models. This problem has been addressed by performing weight-based pruning before ε removal without affecting transliteration performance.

[0065] In some embodiments, given that bilingual word usage distribution is far from uniform, several words reappear very frequently in the training data. To take advantage of this, a cache of successful transliterations is introduced with a maximum size of 100K elements, thereby reducing the transliteration of common terms to table lookups.

[0066] Table 3 presents the contribution of the above optimization steps to speedup / memory reduction. For example, the above optimizations can reduce the overall training time of a language model trained on 280 billion words from 165 hours to 12 hours.

[0067] optimization Speed ​​(milliseconds) Benchmarks 123 + Transliteration transducer clipping 109.0 + Parallel ε removal 72.6 +ε removes previous weight-based clipping 61.6 + Cache frequent transliterations 25.0

[0068] Table 3

[0069] Effect of various optimizations on transliteration speed calculated on an average pronunciation of four words

[0070] The experiments discussed below were conducted on training and test sets that were anonymized and represented Indian voice search traffic with manually transcribed speech. The training set was augmented with a few original transcripts that were artificially corrupted by adding varying degrees of noise and reverberation using a room simulator, so that the overall SNR varied between 0 and 20db. The signal processing pipeline for all languages ​​extracted 80-dimensional log-mel filter bank output features at a standard frame rate of 10ms. The acoustic model for all languages ​​was an LSTM with 5 layers, each consisting of 768 LSTM units. The acoustic models were trained in TensorFlow using asynchronous stochastic gradient descent minimizing the Connectionist Temporal Classification (CTC) and state-level minimum Bayesian risk (sMBR) objective functions. Table 4 presents the amount of training data used in the experiments for each Indian language. The test data varied between 6,000 and 10,000 words. It can be seen that the available data varies greatly between the languages. A detailed analysis of Hindi is presented, as Hindi is the language with the highest number of code switches with English and the largest number of training tokens. The Hindi training dataset consists of approximately 10,000 hours of training data from 10 million utterances. The proposed approach has also been validated on other Indic languages, which typically have 10% to 20% of Hindi data.

[0071]

[0072] Table 4

[0073] Indian language training set

[0074] Table 5 shows the significant difference in the measured WER after correcting for errors related to the writing system. The proposed toWER metric is calculated after transliterating both the reference and the hypothesis into one writing system corresponding to the local language environment. It can be seen that there is a correlation between the percentage of Latin scripts and the proposed metric, which can be used as a good indicator of the extent of code switching in these languages. In languages ​​such as Malayalam, Telugu, Marathi and Urdu, the number of code switching is less than in languages ​​such as Hindi and Bengali, and toWER reflects this. Therefore, transliteration can be a means of correcting writing system errors caused by inconsistencies and a means of separating modeling errors from rendering errors. The transliteration score can reduce the ambiguity caused by code switching and can eliminate ambiguities and transcription errors. Therefore, the proposed toWER is a better metric to evaluate any algorithm improvement.

[0075]

[0076] Table 5

[0077] The impact of toWER on fixing rendering errors when measuring WER on voice search queries in several Indian languages

[0078] Influenced by the transliteration optimization score, methods for normalizing the training data of the language model using transliteration are also explored. First, the results of training a transliteration language model (LM) on Hindi are discussed. All texts in various corpora used to build the Hindi language model are first transliterated to eliminate any Latin scripts. Subsequently, the normalized scripts in Devanagari are used to train the 5-gram LM for the first pass and the class-based maximum entropy model for the second pass. Table 6 presents the results obtained when these models are used to decode two different test sets including voice search queries and dictation data. In order to compare with various writing systems as input to the language model, we define the LM based on Devanagari data only as the LM built using only all utterances containing Devanagari scripts. Any speech containing bilingual text in Devanagari and Latin scripts is not used in the language model construction. As expected, this leads to loss of context modeling, data reduction, and introduces a mismatch between the training and test set distributions. The hypothetical transliteration score produced by this LM repairs the mismatch with the reference transcription (row 2). Keeping data in both writing systems ensures that the context of code switches is preserved, but introduces all the challenges discussed in Section 2, including the occurrence of identical words in both Devanagari and Latin. Since all other data from Latin is included in the LM as is, the mismatch between the reference and the hypothesis increases even further, leading to an artificially inflated conventional WER (row 3). The toWER metric reflects the actual error rate (row 4). It can be seen that retraining the LM using all data transliterated to Devanagari leads to significant gains on the voice search and dictation test sets (row 5). Therefore, building the LM by transliterating all training data to Devanagari, thereby introducing consistent text normalization, leads to a relative improvement of 3% to 8% in WER on both test sets.

[0079]

[0080] Table 6

[0081] Impact of transliteration on fixed modeling and rendering errors for voice search queries and dictation in Hindi using a maximum entropy based language model

[0082] The effect of transliteration on a long short-term memory (LSTM) neural network model was also investigated. As shown in Table 7, training models with transliterated text provided performance gains similar to those of the maximum entropy-based LM for the voice search task, while smaller than the dictation task. Although not surprising, it validates the hypothesis that transliteration-based normalization and scoring for training helps separate modeling errors from rendering errors and helps accurately assess the performance of the model. For the voice search task shown in Tables 6 and 7, it can be concluded that the performance of the LSTM LM and the maximum entropy-based LM are very similar when using regular WER (32.0 vs 31.5), while toWER shows that the maximum entropy-based LM performs better than the LSTM (20.6 vs 22.3). The significance of these gains can actually be measured by human evaluators in a side-by-side comparison study explained below.

[0083] Transliteration can also improve the accuracy of acoustic modeling. In the acoustic model (AM) training on Hindi, various experiments were performed using transliterations of each utterance. All words in the AM training data written in Latin were first transliterated into Devanagari script and the pronunciations were derived using the Hindi phonetic alphabet. After the model was trained to convergence using the CTC criterion, the transliterated AM showed a small improvement in performance over the model trained using both writing systems (see Table 8). It is expected that the improvements will be more pronounced for sMBR training since the numerator and denominator lattices required for sMBR training will always be presented in Devanagari script.

[0084]

[0085] Table 7

[0086] The impact of transliteration on ASR performance of voice search queries and dictated utterances for LSR LM

[0087]

[0088] Table 8

[0089] Impact of the proposed method on acoustic modeling

[0090] Next, Table 9 below presents the impact of the proposed approach on several other Indian languages. Except for Malayalam and Tamil, all other languages ​​achieve significant and consistent gains. This can be attributed to the number of Latin words present in the training corpus. For these two languages, the Figure 2As can be seen in Table 10, the voice search corpus containing spoken queries contains almost no Latin, while the corpus containing web-based queries contains more Latin. However, the web-based corpus receives a very low interpolation weight for this task, so it has little impact on WER. A similar trend is observed for the dictation task with the transliterated LM (see Table 10, to WER is reduced by up to 10% relative).

[0091]

[0092] Table 10

[0093] ASR performance with transliteration LM for dictation tasks in various Indian languages

[0094] Figure 3 is a graph showing WER values, toWER values, and correlation with the code-switching percentage measured as the percentage of Latin in the data.

[0095] Transliteration can lead to both positive and negative changes in the recognition results. Table 11 shows several examples where the conventional WER metric artificially increases the error. In the first example, the utterance of Satta Matka is transcribed in Latin script, while the ASR system assumes it in Sanskrit, so it counts as two substitution errors. However, since toWER transliterates it to Sanskrit before calculating the error rate, it does not produce any errors. A similar situation can be seen in the second example for the word Discovery.

[0096]

[0097] Table 11

[0098] Sample bug fixed by toWER

[0099] However, not all issues related to code switching can be resolved by transliteration alone. Some such instances are highlighted in Table 12, where the transliteration process introduced errors that did not exist before. In the first example, an utterance in Latin was read as the entire movie Tiger zinda hai. The reference contained the first three words in Latin and the last two words in Sanskrit. By design, the ASR assumed Sanskrit. The result of transliterating the reference and the assumption into the common Sanskrit writing system resulted in the error of Zinda vs Jinda. Similarly, in the second example, the reference was transliterated as Jamuna, while the assumption produced Jumna, a result of ambiguity in the transliteration process, where either form is acceptable. The third example produced a more classic error. The utterance read in Latin as BA first year timetable. Note that in this example, the transcriber was consistent only in producing the text in Sanskrit. The ASR system correctly assumed the utterance, but incorporated the writing system and calculated three substitution errors based on the WER metric. During the transliteration of the hypothesis, BA was mapped to the Sanskrit Ba (pronounced as bah in the word "bar") and lost the "A" after the abbreviation BA. This will lead to a substitution error using the toWER metric. Overall, despite the above-mentioned prominent cases, the proposed metric does reflect the performance of the system more accurately compared to the traditional WER.

[0100]

[0101] Table 12

[0102] Sample errors introduced by toWER

[0103] As an additional evaluation of the proposed approach, several “side-by-side” (SxS) experiments were conducted, where each anonymized test utterance was automatically transcribed by two ASR systems (baseline vs. transliterated LM). If the two hypotheses differed, they were presented to human evaluators. SxS experiments accurately measure semantic variation rather than minor lexical differences. We also conducted SxS experiments for the case where the hypotheses were generated by a single ASR system, but the evaluators saw the original hypotheses in multiple writing systems as well as in a single native writing system (Devanagari in this case). In Table 13, the following results are presented for the 500 differently pronounced utterances: (1) Variation: the percentage of utterances for which the two systems produced different transcriptions. (2) Gain / Loss: the ratio of gains to losses in the experimental system compared to the baseline. p-values ​​<5% were considered statistically significant.

[0104] It is evident from Table 13 that human evaluators gave a neutral rating to the transliteration hypothesis compared to the hypothesis based on a mixed writing system. This is not unexpected since the semantic content of the two systems being compared has not changed. However, toWER can eliminate rendering errors and provide a better perspective. Similarly, the second row compares two LMs, the baseline system (row 2 in Table 6, toWER of 20.6%) and the system with the transliteration LM (row 4 in Table 6, toWER of 17.2%). For the transliteration LM (experimental system) there are more gains than losses.

[0105]

[0106] Table 13

[0107] Human evaluators’ judgments on toWER and transliterated LM results

[0108] All Indic languages ​​show a correlation between the amount of text written in Latin and the gains obtained using the transliterated LM and toWER metrics on the voice search task. However, in Kannada, performance on the dictation task degrades. To better understand the source of these errors, grapheme error rates were generated for Kannada in the transliterated space and compared to languages ​​such as Bengali, which shows clear gains in transliterated LM.

[0109] Interestingly enough, from Table 14, it can be seen that the number of deletion errors between the baseline and the transliterated LM is much higher in Kannada (30% relative increase) than in Bengali (which remains unchanged). There is also a relative increase of about 3% in substitution errors in Kannada, while there is a relative decrease of 6.4% in substitution errors in Bengali. However, while the grapheme error rate for Bengali is reduced from 18.7% to 17.6% using the transliterated LM, the grapheme error rate for Kannada is only reduced from 10.27% to 10.23%. A close look at the hypothesis indicates that many of the errors are either introduced by transliteration when two words are merged into one, or due to ambiguity in the language where both the merged and split forms are considered correct. The combination of these factors ultimately leads to the degradation of toWER. The improvement in the grapheme error rate is a good indication that the transliterated LM is still useful. It is likely that some of the errors introduced by the transliteration process can be corrected by training the model on matching data.

[0110]

[0111] Table 14

[0112] Grapheme error rates, deletion and substitution errors in Bengali and Kannada dictation tasks. All figures are in %.

[0113] In short, conventional word error rate (WER) metrics are inadequate to measure performance on code-mixed languages ​​due to ambiguities in transcriptions, spelling errors, and borrowing words from two different writing systems. Modeling errors can be precisely measured using the proposed transliteration-based “toWER” metric, which eliminates rendering errors. Consistent normalization of training transcriptions for language and acoustic modeling can provide significant gains of up to 10% over several code-switched Indian languages ​​used in voice search and dictation operations. Significant gains can be achieved using a simple transliteration-based approach to consistently standardize training data and accurately measure model robustness and accuracy.

[0114] The embodiments of the present invention and all functional operations described in this specification may be implemented in digital electronic circuits or in computer software, firmware or hardware, including the structures disclosed in this specification and their equivalents or one or more combinations thereof. Embodiments of the present invention may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium to be executed by a data processing device or to control the operation of a data processing device. The computer-readable medium may be a non-transitory computer-readable storage medium, a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagation signal, or a combination of one or more thereof. The computer-readable medium may be a transient medium, such as an electrical, optical or electromagnetic signal. The term "data processing device" encompasses all devices, apparatuses and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device.

[0115] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to be executed on one computer or multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0116] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0117] For example, processors suitable for executing computer programs include general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, the computer will also include or be operably coupled to receive data from one or more mass storage devices for storing data, such as magnetic, magneto-optical disks or optical disks, or transfer data to one or more mass storage devices or both. However, the computer does not have to have such devices. In addition, the computer can be embedded in another device, such as a tablet computer, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a global positioning system (GPS) receiver, etc. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0118] To provide interaction with a user, embodiments of the present invention may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, a keyboard, and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input.

[0119] Embodiments of the invention may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the invention, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0120] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between client and server is created by computer programs running on the respective computers and having a client-server relationship to each other.

[0121] Although this specification contains many details, these details should not be interpreted as limitations on the scope of the invention or the scope of the claimed protection, but should be interpreted as descriptions of features specific to specific embodiments of the invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. On the contrary, various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination. Moreover, although the features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be cut out from the combination, and the claimed combination may be used in a sub-combination or sub-combination variation.

[0122] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0123] In each instance where an HTML file is mentioned, other file types or formats may be substituted. For example, an HTML file may be replaced with XML, JSON, plain text, or other types of files. Also, where a table or hash table is mentioned, other data structures (such as a spreadsheet, a relational database, or a structured file) may be used.

[0124] Thus far, certain embodiments of the present invention have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Claims

1. A method performed by one or more computers, the method comprising: accessing, by the one or more computers, a data set indicating language examples for a first script, wherein at least some of the language examples include words in the first script and words in one or more other scripts; transliterating, by the one or more computers, at least a portion of some of the language examples into the first script to generate a training data set having words transliterated into the first script; and generating, by the one or more computers, a speech recognition model based on occurrences of sequences of words in the training data set having words transliterated to the first script, Wherein, transliterating the language example includes: accessing a blacklist of terms in scripts other than the first script; and Transliterations of instances of terms from the blacklist that appear in the language sample are bypassed.

2. The method according to claim 1, wherein: The speech recognition model is a language model, an acoustic model, a sequence-to-sequence model or an end-to-end model.

3. The method according to claim 1, wherein: Transliteration involves mapping different glyphs representing text from different scripts into a single standardized transliterated representation.

4. The method according to claim 1, wherein: Transliterating the language example includes transliterating words in the language example that are not in the first script into the first script.

5. The method according to claim 1, wherein: Words written in the second script are individually transliterated into the first script on a word-by-word basis.

6. The method according to claim 1, further comprising: determining a test set of language examples for testing the speech recognition model; generating a standardized test set by transliterating words in the language examples in the test set that are not written in the first script into the first script; Obtaining an output of a speech recognition model corresponding to a language example in the test set; normalizing the output of the speech recognition model by transliterating words output by the speech recognition model that are not written in the first script to the first script; and Based on a comparison of the standardized test set with a standardized speech recognition model output, an error rate of the speech recognition model is determined.

7. The method according to claim 6, wherein: The error rate is a word error rate, and wherein the method comprises based on the word error rate: Determining whether to continue training the speech recognition model or terminate the training of the speech recognition model; Changing the training data set used to train the speech recognition model; Setting the size, structure or other characteristics of the speech recognition model; or Select one or more speech recognition models for the speech recognition task.

8. The method of claim 1 further comprising determining a modeled error rate for the speech recognition model, wherein acoustically similar words written in any of a plurality of scripts are accepted as correct transcriptions without penalizing the output of words in a script that differs from a corresponding word in a reference transcription.

9. The method of claim 8 further comprising determining a rendering error rate of the speech recognition model, the rendering error rate being a measure of the difference between the scripts of words in the output of the speech recognition model relative to the scripts of corresponding words in a reference transcription.

10. The method according to claim 1, wherein: The transliteration is performed using a finite state transducer network trained to perform transliteration into the first script.

11. The method according to claim 1, wherein: Transliterating includes, for at least one language example, performing multiple rounds of transliteration between scripts to arrive at a transliterated representation in the first script included in a training data set in the first script.

12. The method of claim 1, further comprising determining a score indicative of a degree of mixing of scripts in the language sample; and Based on the ratings stated: selecting parameters for tailoring finite state transducer networks for transliteration; selecting parameters for tailoring the speech recognition model; or A size or structure is selected for the speech recognition model.

13. The method according to claim 1, wherein: Generating the speech recognition model includes: After transliterating at least a portion of some language examples into the first script, determining, by the one or more computers, a number of occurrences of different word sequences in the training data set in the first script; and A speech recognition model is generated by the one or more computers based on the number of occurrences of different word sequences in the training data set in the first script.

14. The method according to claim 1, wherein: The speech recognition model includes a recursive neural network, and generating the speech recognition model includes training the recursive neural network.

15. The method of any one of claims 1 to 14, further comprising performing speech recognition on an utterance by the one or more computers using the speech recognition model.

16. A method for performing speech recognition, comprising: receiving, by one or more computers, audio data representing speech; and mapping, by the one or more computers, the audio data to text representing the utterance using the speech generation model, The speech recognition model is generated according to the method according to any one of claims 1 to 14.

17. A system comprising: one or more computers; and One or more computer-readable media storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of the method according to any one of claims 1 to 16.

18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method according to any one of claims 1 to 16.

Images