Multilingual rescoring models for automatic speech recognition

Abstract

The method (400) receives a sequence of acoustic frames (110) extracted from speech data corresponding to an utterance (106). During a first pass (301), the method processes the sequence of acoustic frames to generate N candidate hypotheses (204) for the utterance. During a second pass (302), and for each candidate hypothesis, the method comprises: generating a respective unnormalized likelihood score (325), generating a respective external language model score (315), generating a silico score (205) that models the prior statistics of the corresponding candidate hypothesis, and generating a respective overall score (355) for the candidate hypothesis based on the unnormalized likelihood score, the external language model score, and the silico score. The method also selects the candidate hypothesis from the N candidate hypotheses having the highest respective overall score as the final transcription (120) of the utterance.

Description

[Technical field]

[0001] The present disclosure relates to multilingual rescoring models for automatic speech recognition. [Background technology]

[0002] Automatic speech recognition (ASR) systems provide technology that is commonly used on mobile devices and other devices. In general, automatic speech recognition ASR systems attempt to provide an accurate transcription of what a user speaks into a mobile device. More specifically, an automatic speech recognition ASR system generates multiple transcription candidates and outputs the transcription candidate that is most likely to match the speech input. In some cases, an automatic speech recognition ASR system outputs an inaccurate transcription that does not match what the user actually spoke. In such cases, the automatic speech recognition ASR system rescores the multiple transcription candidates and outputs an accurate transcription that matches the speech input. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] OGAWA ATSUNORI ET AL, "Rescoring N-Best Speech Recognition List Based on One-on-One Hypothesis Comparison Using Encoder-Classifier Model", 2018 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), IEEE,15 April 2018 (2018-04-15), page 6099-6103,XP033403971,DOI: 10.1109 / ICASSP.2018.8461405 Summary of the Invention [Problem to be solved by the invention]

[0004] However, one challenge with rescoring is that ASR systems rely on linguistic information from the speech input to accurately rescore multiple transcription candidates, which makes rescoring in a multilingual speech environment often a tedious task. [Means for solving the problem]

[0005] One aspect of the disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations using a multilingual rescoring model for automatic speech recognition. The operations include receiving a sequence of acoustic frames extracted from audio data corresponding to an utterance. During a first pass, the operations include processing the sequence of acoustic frames using the multilingual speech recognition model to generate N candidate hypotheses for the utterance. During the second pass, for each of the N candidate hypotheses, the method includes: generating a respective unnormalized likelihood score based on the sequence of acoustic frames and the corresponding candidate hypothesis using a neural oracle search (NOS) model, generating a respective external language model score using the language model, generating a respective stand-alone score that models prior statistics of the corresponding candidate hypothesis generated during the first pass, and generating an overall score for each of the candidate hypotheses based on the unnormalized score, the external language model score, and the stand-alone score. The operations also include selecting the candidate hypothesis having the highest respective overall score from among the N candidate hypotheses as the final transcription of the utterance.

[0006] The embodiments of the present disclosure may include one or more of any of the following features: In some implementations, each candidate hypothesis of the N candidate hypotheses includes a respective sequence of word or subword labels, where each word or subword label is represented by a respective embedding vector. The external language model may be trained with text-only data. In some examples, the neural oracle search NOS model includes a language-specific neural oracle search NOS model. In these examples, the operations further include receiving a language identifier indicating a language of the utterance, and selecting the language-specific neural oracle search NOS model from among a plurality of language-specific neural oracle search NOS models, each trained for a different language.

[0007] Optionally, the neural oracle search NOS model may comprise a multilingual neural oracle search NOS model. In some implementations, the external language model comprises a language-specific external language model. In these implementations, the operations further comprise receiving a language identifier indicating a language of the utterance and selecting a language-specific external language model from among a plurality of language-specific external language models, each trained in a different respective language. The neural oracle search NOS model may comprise two unidirectional long short-term memory (unidirectional LSTM) layers. In some examples, the speech recognition model comprises an encoder-decoder architecture comprising a conformer encoder having a plurality of conformer layers and an LSTM decoder having two LSTM layers.

[0008] Another aspect of the disclosure provides a system comprising data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a sequence of acoustic frames extracted from audio data corresponding to an utterance. During a first pass, the operations include processing the sequence of acoustic frames using a multilingual speech recognition model to generate N candidate hypotheses for the utterance. During a second pass, for each of the N candidate hypotheses, the method includes: generating a respective unnormalized likelihood score based on the sequence of acoustic frames and the corresponding candidate hypothesis using a neural oracle search (NOS) model; generating a respective external language model score using a language model; generating a simplex score that models a priori statistics of the corresponding candidate hypothesis generated during the first pass; and generating an overall score for each of the candidate hypotheses based on the unnormalized score, the external language model score, and the simplex score. The operations also include selecting the candidate hypothesis from among the N candidate hypotheses having the highest respective overall score as the final transcription of the utterance.

[0009] Embodiments of the present disclosure may include one or more of any of the following features: In some implementations, each candidate hypothesis of the N candidate hypotheses comprises a respective sequence of word or subword labels, where each word or subword label is represented by a respective embedding vector; The external language model may be trained with text-only data; In some examples, the neural oracle search NOS model comprises a language-specific neural oracle search NOS model; In these examples, the operations further include receiving a language identifier indicating a language of the utterance, and selecting the language-specific neural oracle search NOS model from among a plurality of language-specific neural oracle search NOS models, each trained for a respective different language.

[0010] Optionally, the neural oracle search NOS model may comprise a multilingual neural oracle search NOS model. In some implementations, the external language model comprises a language-specific external language model. In these implementations, the operations further comprise receiving a language identifier indicating a language of the utterance and selecting a language-specific external language model from among a plurality of language-specific external language models, each trained in a different respective language. The neural oracle search NOS model may comprise two unidirectional long short-term memory (unidirectional LSTM) layers. In some examples, the speech recognition model comprises an encoder-decoder architecture comprising a conformer encoder having a plurality of conformer layers and an LSTM decoder having two LSTM layers.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of an audio environment in which an example speech recognition model is implemented. [Diagram 2] 2 is a schematic diagram of the exemplary speech recognition model of FIG. 1; [Figure 3A] Schematic diagram of an exemplary rescoring process using multiple language-specific Neural Oracle Search (NOS) models. [Figure 3B] Schematic diagram showing an example of a rescoring process using a multilingual neural oracle search NOS model. [Figure 4] 1 is a flowchart of an example arrangement of method operations using a multilingual rescoring model for automatic speech recognition. [Diagram 5] 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference numbers in the various drawings indicate like elements. Automatic speech recognition (ASR) systems are becoming increasingly prevalent in user devices to provide more accurate transcriptions of what a user has said. However, automatic speech recognition ASR systems may generate inaccurate transcriptions that misrecognize what a user actually said. In some configurations, an automatic speech recognition ASR system generates N best candidate hypotheses for a spoken utterance and outputs the best candidate hypothesis as the final transcription. However, the configuration of N best candidate hypotheses results in a word error rate (WER) that is nearly 50% lower than the configuration of a single best hypothesis. Therefore, in some implementations, an automatic speech recognition ASR system rescores the N best candidate hypotheses by integrating additional information to increase the word error rate (WER). Such rescoring implementations rely on language information (i.e., a language identifier spoken by the user) in a multilingual speech environment and provide only a small improvement in the word error rate (WER). The challenges discussed above have revealed a gap in word error rate (WER) performance between an ASR system that uses N best candidate hypothesis configurations and an ASR system that uses a single best candidate hypothesis configuration.

[0014] Accordingly, implementations herein are directed to a method and system that performs a rescoring process that generates N candidate hypotheses for a corresponding utterance, and selects the most likely candidate hypothesis to output as a final transcription. In particular, during a first pass, the rescoring process generates N candidate hypotheses by using a multilingual speech recognition model. Then, during a second pass, the rescoring process generates, for each candidate hypothesis, a respective unnormalized likelihood score by using a neural oracle search (NOS) model, an external language model score, and a standalone score that models the candidate hypothesis' prior statistics. As will be made clear below, the neural oracle search NOS model may be a language-specific neural oracle search NOS model or a multilingual neural oracle search NOS model. Furthermore, during the second pass, the rescoring process generates an overall score for each candidate hypothesis based on the unnormalized likelihood score, the external language model score, and the standalone score. The rescoring process selects the candidate hypothesis with the highest overall score as the final transcription for the utterance.

[0015] 1 is an example of an audio (speech) environment 100. In the audio environment 100, the way in which a user 104 interacts with a computing device, such as a user device 10, may be through voice input. The user device 10 is configured to capture sounds (e.g., streaming audio data) from one or more users 104 in the audio environment 100. Here, streaming audio data may refer to speech (spoken words, voice utterances) 106 by the user 104 that function as audible queries, commands to the user device 10, or audible communications captured by the user device 10. The voice-enabled system of the user device 10 may field the queries or commands by responding to the queries and / or having the commands executed / fulfilled by one or more downstream applications.

[0016] The user device 10 may correspond to any computing device associated with a user 104 and capable of receiving acoustic data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 comprises data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12. The memory hardware 14 stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further comprises an audio system 16. The audio system 16 includes audio capture devices (e.g., microphones) 16, 16a for capturing and covering speech 106 in the audio environment 100 and converting it into electrical signals, and audio output devices (e.g., speakers) 16, 16b for communicating audible audio signals (e.g., as output audio data from the user device 10). Although the user device 10 implements a single audio (sound) capture device 16a in the illustrated example, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure. Thereby, one or more audio capture devices 16a in the array may not be physically present on the user device 10, but may be in communication with the audio (sound) system 16.

[0017] In the speech environment 100, an automatic speech recognition (ASR) system 118 implementing a speech recognition model (i.e., an automatic speech recognition ASR model) 200 resides on a user device 10 of a user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system running in a cloud computing environment) that communicates with the user device 10 via a network 40. The automatic speech recognition ASR system 118 may also implement one or more external language models 310 and a neural oracle search (NOS) model 320. The user device 10 and / or the remote computing device (i.e., the remote server) 60 also include a speech subsystem 108. The speech subsystem 108 is configured to receive an utterance 106 spoken by the user 104 and captured by the speech capture device 16a, and to convert the utterance 106 into a corresponding digital format associated with an input acoustic frame (acoustic frame 110) that can be processed by the automatic speech recognition ASR system 118. In the illustrated example, while the user speaks each utterance 106, the speech subsystem 108 converts the utterance 106 into corresponding speech data (e.g., acoustic frames 110) for input to the automatic speech recognition ASR system 118. The speech recognition model 200 then receives as input the speech data (110) corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (e.g., speech recognition result / hypothesis) of the utterance 106. As described in more detail below, the speech recognition model 200 may comprise an end-to-end speech recognition model 200 trained with variable look-ahead speech context to enable the speech recognition model 200 to set different durations of look-ahead speech context during inference when performing speech recognition depending on how sensitive the query specified by the utterance 106 is to latency and / or how tolerant the user 104 is to latency.For example, the digital assistant application 50 running on the user device 10 can request speech recognition depending on how sensitive the query specified by the utterance 106 is to latency and / or how much latency the user 104 has.

[0018] In some implementations, the speech recognition model 200 performs streaming speech recognition on the speech data 110 during a first pass to generate N candidate hypotheses 204 (FIGS. 3A, 3B). The neural oracle search NOS model 320 and the language model 310 rescore the N candidate hypotheses 204 during a second pass to generate a final transcription 120. For example, in the illustrated example, the speech recognition model 200 performs streaming speech recognition on the speech data 110 to generate partial speech recognition results (i.e., partial transcriptions) 120, 120a (based on the N candidate hypotheses 204). The language model 310 and the neural oracle search NOS model 320 rescore the N candidate hypotheses 204 to generate a final speech recognition result (i.e., final transcription) 120, 120b. In particular, the speech recognition model 200 may use a look-ahead speech context, which may be set to zero (or about 240 milliseconds), to generate the partial speech recognition result 120a. Thus, the final speech recognition result 120b for the input utterance (utterance 106) may lag behind the partial speech recognition result 120a for the input utterance.

[0019] The user device 10 and / or the remote computing device 60 also execute a user interface generator 109 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 109 can display the partial speech recognition results 120a in a streaming manner during a first time 1. The user interface generator 109 can then display the final speech recognition results 120b during a second time 2. In some configurations, the transcription 120 output from the automatic speech recognition ASR system 118 is processed by, for example, a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60 to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) may convert the transcription into synthetic speech for audible output by the user device 10 and / or another device.

[0020] In the illustrated example, a user 104 is communicating with a digital assistant application 50. The digital assistant application 50 displays a digital assistant interface 18 on the screen of the user device 10 to depict a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is an utterance 106 that is captured by the voice capture device 16a and processed by the voice system 16 of the user device 10. In this example, the voice system 16 receives the utterance 106 and converts the utterance 106 into acoustic frames 110 for input to an automatic speech recognition ASR system 118.

[0021] Continuing with this example, as the speech recognition model 200 receives acoustic frames (i.e., audio data) 110 corresponding to an utterance 106 spoken by the user 104, it not only encodes the acoustic frames 110 but also decodes the encoded acoustic frames 110 into partial speech recognition results 120a. During a first time 1, the user interface generator 109 presents, via the digital assistant interface 18, a representation of the partial speech recognition results 120a of the utterance 106 to the user 104 of the user device 10 in a streaming manner, with words, word fragments, and / or individual characters appearing on the screen as soon as they are spoken.

[0022] During the second pass, and after all acoustic frames 110 corresponding to the utterance 106 have been received, the automatic speech recognition ASR system 118 uses the language model 310 and the neural oracle search NOS model 320 to rescore each of the N candidate hypotheses 204 and select from the N candidate hypotheses 204 the candidate hypothesis 204 that has the highest likelihood of being an accurate transcription 120 of the utterance 106. During the second time 2, the user interface generator 109 presents a representation of the final speech recognition result 120b of the utterance 106 to the user 105 of the user device 10 via the digital assistant interface 18. In some implementations, the user interface generator 109 replaces the representation of the partial speech recognition result 120a with the representation of the final speech recognition result 120b. For example, the final speech recognition result 120b is estimated to be more accurate than the partial speech recognition result 120a that was generated without utilizing look-ahead speech context. Thus, the final speech recognition result 120b, which is finally displayed as the transcription 120, can correct terms that may have been misrecognized in the partial speech recognition result 120a. In this example, the streaming partial speech recognition result 120a is output by the speech recognition model 200 and displayed on the screen of the user device 10 at a first time 1, and is associated with low latency, thus providing the user 104 with a sense of responsiveness that their query is being processed. On the other hand, the final speech recognition result 120b, which is displayed on the screen at a second time 2, improves the speech recognition quality in terms of accuracy, but at the expense of increased latency. However, since the partial speech recognition result 120a is displayed as the user speaks the utterance 106, the high latency associated with generating and ultimately displaying the final recognition result is not noticeable to the user 104.

[0023] In the example shown in FIG. 1, the digital assistant application 50 can respond to a question posed by the user 104 by using natural language processing. Natural language processing generally refers to interpreting written language (e.g., partial speech recognition result 120a and / or final speech recognition result 120b) and determining whether the written language prompts an action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from the user 104 is about the user's schedule, and more specifically, about a concert on the user's schedule. By recognizing these details with natural language processing, the automated assistant returns a response 19 to the user's query. The response 19 states, "The venue doors open at 6:30 p.m. and the concert starts at 8 p.m." In some configurations, the natural language processing is performed on a remote server 60 in communication with the data processing hardware 12 of the user device 10.

[0024] Referring to FIG. 2, an exemplary frame alignment based transducer model 200a comprises a recurrent neural network-transducer (RNN-T) model architecture that adheres to latency constraints associated with interactive applications. The use of the recurrent neural network-transducer RNN-T model architecture is exemplary, and the frame alignment based transducer model 200 may include other architectures, such as transformer-transducer and conformer-transducer model architectures, among others. Because the recurrent neural network-transducer RNN-T model 200 provides a small computational footprint and utilizes less memory requirements than conventional automatic speech recognition ASR architectures, the recurrent neural network-transducer RNN-T model architecture is suitable for performing speech recognition entirely on the user device 102 (e.g., no communication with a remote server is required). The recurrent neural network-transducer RNN-T model 200 comprises an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210 may comprise a recurrent network of stacked long-short-term (LSTM) layers, roughly analogous to an acoustic model (AM) in a conventional automatic speech recognition (ASR) system. For example, the encoder may receive a sequence of d-dimensional feature vectors, e.g., an acoustic frame 110 (FIG. 1) x=(X1, X2, ..., X T ), where Xt∈R d At each output step, a high-level feature representation is generated by reading the input data (h1, R is a white-spaced character). enc ,…,h T enc It is written as follows.

[0025] Similarly, the prediction network 220 is also an LSTM network (i.e., an LSTM decoder) that, like the language model (LM), receives the non-blank symbol sequence (i.e., label history) 245 (y0,...,y ui-1 ) into a dense representation p ui Finally, in the recurrent neural network-transducer RNN-T model architecture, the representations generated by the encoder and predictor / decoder networks 210, 220 are combined by a combination network 230. The prediction network 220 may be replaced by an embedding lookup table to improve latency by outputting a looked-up sparse embedding instead of processing dense representations. The combination network then calculates P(y i |X ti ,y0,…,y ui-1) in the output step (e.g., time step). In other words, the combination network 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step), where a "possible speech recognition hypothesis" corresponds to a set of output labels, each representing a symbol / character in the specified natural language. For example, if the natural language is English, the set of output labels may comprise 27 symbols, e.g., one label for each of the 26 characters in the English alphabet and one label specifying a space. Thus, the combination network 230 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. The set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters and potentially punctuation or other symbols), but the set of output labels is not so limited. For example, the set of output labels may comprise word fragments and / or whole words in addition to or instead of graphemes. The output distribution of the combination network 230 can comprise a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output yi of the combination network 230 can comprise 100 different probability values, one probability value for each output label. The probability distributions can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in a beam search process (e.g., by the softmax layer 240) to determine the transcription 120.

[0026] The softmax layer 240 may employ any technique to select the output label / symbol that has the highest probability in the distribution as the next output symbol predicted by the recurrent neural network-transducer RNN-T model 200 at the corresponding output step. In this way, the recurrent neural network-transducer RNN-T model 200 does not make the assumption of conditional independence, but rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The recurrent neural network-transducer RNN-T model 200 assumes that the output symbols are independent of future acoustic frames 110. This allows the recurrent neural network-transducer RNN-T model to be employed in a streaming manner.

[0027] In some examples, the encoder network (e.g., acoustic encoder) 210 of the recurrent neural network-transducer RNN-T model 200 is an encoder-decoder architecture having a conformer-based encoder with a stack of conformer layers, where each conformer layer includes a series of multi-headed self-attention layers, depth-wise convolutional layers, and feed-forward layers. In some examples, the conformer-based encoder can include a stack of 17 conformer layers. The encoder network 210 may include other types of encoders with multi-headed self-attention mechanisms. For example, the encoder network 210 may be a transformer-based encoder or a lightweight convolutional (LConv)-based encoder. The encoder network 210 may also be RNN-based with a sequence of LSTM layers. The prediction network 220 may be an LSTM decoder with two 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Alternatively, the prediction network 220 may include a stack of transformer or conformer blocks, or an embedded lookup table, instead of an LSTM layer. Finally, the combination network 230 may also have 640 hidden units. The softmax layer 240 may consist of a unified set of word pieces or graphemes that have been generated using all unique word pieces or graphemes in multiple training datasets.

[0028] 3A and 3B, in some implementations, the remote server 60 (FIG. 1) performs an exemplary rescoring process 300 for rescoring the N candidate hypotheses 204 generated by the automatic speech recognition ASR model 200 during a first pass 301. Alternatively, the user device 10 (FIG. 1) may perform the exemplary rescoring process 300 in addition to or instead of the remote server 60 (FIG. 1). The rescoring process 300 may include rescoring a sequence of acoustic frames 110 (X1, X2, ..., X T ), N candidate hypotheses 204, 204a to 204n (H1, H2, ..., H N The rescoring process 300 further comprises a second pass 302 for rescoring each of the N candidate hypotheses 204 by integrating additional information resources, which will be described in more detail below. Thus, the second pass 302 comprises a sequence classification object configured to select, from the N candidate hypotheses 204, the candidate hypothesis 204 that has the highest likelihood of being an accurate transcription of the utterance 106.

[0029] In particular, the automatic speech recognition ASR model 200 receives a sequence of acoustic frames 110 extracted from speech data corresponding to the utterance 106. During a first pass 301, the automatic speech recognition ASR model 200 processes the sequence of acoustic frames 110 to generate N candidate hypotheses 204 for the utterance 106, where each candidate hypothesis 204 corresponds to a candidate transcription 120 of the utterance 106 and is represented by a respective sequence of words, subwords, and / or grapheme labels represented by a respective embedding vector. Each candidate hypothesis 204 further comprises a simplex score 205 that models a priori statistics of the corresponding candidate hypothesis 204. That is, the simplex score 205 may indicate a confidence that the corresponding candidate hypothesis 204 is an accurate transcription of the utterance 106. The confidence of the simplex score 205 may also indicate a frequency of the utterance 106 previously realized (e.g., the number of times the candidate hypothesis 204 has been previously uttered).

[0030] The automatic speech recognition ASR model 200 may generate any number of candidate hypotheses 204 (e.g., N may be any integer value). In some examples, the automatic speech recognition ASR model 200 outputs a specified number of candidate hypotheses 204 based on predefined parameters. For example, the automatic speech recognition ASR model 200 outputs five candidate hypotheses 204 (i.e., N=5) for every utterance 106. For example, the N candidate hypotheses 204 may correspond to an N-best list of candidate hypotheses associated with the N candidate hypotheses having the highest simplex scores 205. In other examples, the automatic speech recognition ASR model 200 outputs all candidate hypotheses 204 having simplex scores 205 that meet a threshold.

[0031] In the illustrated example, the automatic speech recognition ASR model 200 processes a sequence of acoustic frames 110 corresponding to the utterance 106 "PLAY_NEXT_SONG" previously spoken by the user 104. The automatic speech recognition ASR model 200 then generates three candidate hypotheses 204 (i.e., N=3). That is, the plurality of candidate hypotheses 204 includes "PLAY_NEXT_SONG," which has a standalone score 205 of 0.6, "HEY_NEXT_LONG," which has a standalone score 205 of 0.3, and "PLAY_NEXT_PONG," which has a standalone score 205 of 0.8. Here, the rescoring process 300 may output the candidate hypothesis 204 "PLAY_NEXT_PONG" (play the next pong) as the partial transcription 120a (FIG. 1) because it has the highest unitary score 205. Alternatively, the rescoring process 300 may refrain from outputting the partial transcription until the rescoring process has produced a final transcription. In this particular example, the candidate hypothesis 204 with the highest unitary score 205 is an inaccurate transcription of the utterance 106 already spoken by the user 104.

[0032] The automatic speech recognition ASR model 200 may be a multilingual automatic speech recognition ASR model configured to recognize utterances 106 spoken in multiple languages. That is, the single automatic speech recognition ASR model 200 can receive an utterance 106 in a first language to generate N candidate hypotheses 204 in a first language, and can receive another utterance 106 in a different second language to generate N candidate hypotheses 204 in a second language. Furthermore, the single automatic speech recognition ASR model can receive an utterance 106 comprising a code-mixed speech comprising terms of both the first language and the second language. In this manner, the rescoring process 300 can be implemented in a multilingual speech environment with the single multilingual automatic speech recognition ASR model 200.

[0033] In some implementations, the second pass 302 may receive the N candidate hypotheses 204 from the first pass 301 and generate a corresponding overall score 355 by integrating additional information for each candidate hypothesis 204. The overall score 355 may indicate a more accurate confidence that each candidate hypothesis 204 is an accurate transcription than the individual scores 205 from the first pass 301. The second pass 302 may then select the candidate hypothesis 204 having the highest overall score 355 as the transcription 120 (i.e., the final transcription 120b (FIG. 1)).

[0034] More specifically, during the second pass 302, an external language model (LM) 310 receives the N candidate hypotheses 204 to generate a respective external language model score 315 for each candidate hypothesis 204. In some implementations, the external language model LM 310 comprises a recurrent neural network language model RNN_LM. Here, the external language model LM 310 may comprise multiple language-specific external language models LM 310, 310a-310n, each trained on text-only data (i.e., unpaired data) of a particular language. As such, the "external language model LM 310" and the "language-specific external language model LM 310" may be used interchangeably herein. Thus, each language-specific external language model LM 310 is configured to generate an external language model score (i.e., a language model score) 315 for the utterance 106 in the respective language. For example, a first language-specific external language model LM 310, 310a trained with English text-only data generates a language model score 315 for an utterance 106 spoken in English. A second language-specific external language model LM 310, 310b trained with Spanish text-only data generates a language model score 315 for an utterance 106 spoken in Spanish. Multiple external language models LM 310 can be trained in any number of languages, where each external language model LM 310 is trained with text-only data in a different respective language.

[0035] Thus, the external language model LM310 can select the language-specific external language model LM310 corresponding to the language of the utterance 106 from among the multiple language-specific external language models LM310 by receiving the language identifier 107 indicating the language of the utterance 106. In other words, the rescoring process 300 can select the language-specific external language model LM310 based on the language identifier 107. In some examples, the automatic speech recognition ASR model 200 determines the language identifier 107 based on processing the sequence of acoustic frames 110 of the utterance 106. In other examples, the automatic speech recognition ASR model 200 obtains the language identifier 107 from an external source. For example, a user can configure the automatic speech recognition ASR model for a particular language. In other examples, the automatic speech recognition ASR model 200 can determine the identity of the user 104 who spoke the utterance 106 and identify the language identifier 107 based on the language associated with the identified user 104.

[0036] Thus, during the second pass 302, the rescoring process 300 selects an external language model LM 310 corresponding to the language of the utterance 106 based on the language identifier 107 and generates a language model score 315 for each candidate hypothesis 204. The language model score 315 indicates the likelihood that a sequence of hypothesis terms from the candidate hypotheses 204 will be uttered by the user 104. For example, the external language model LM 310 generates a higher language model score 315 for the candidate hypothesis 204 "What's the weather today?" (Wat_is_the_weather_today) as opposed to the candidate hypothesis 204 "What's the weather? Hooray?" (Wat_is_the_weather_hooray). In particular, the external language model LM 310 generates a higher language model score 315 for "What's the weather today?" (Wat_is_the_weather_today). This is because this sequence of hypothetical terms is likely to appear more frequently in the text-only training data than "What's the weather? Hooray."

[0037] The exemplary rescoring process 300 also includes a neural oracle search (NOS) model 320. The neural oracle search NOS model 320 receives the N candidate hypotheses 204, the sequence of acoustic frames 110, and a label history 245 (e.g., previously output words, word fragments, and / or graphemes). 0:i-1 ) may be output by the automatic speech recognition ASR model 200, the second pass 302 of the rescoring process 300 (e.g., via a rescorer 350), or a combination thereof. In some examples, the label history 245 comprises a transcription of a previous utterance 106 spoken by the user 104. For example, the user 104 may have previously spoken a previous utterance 106 of "Do I have any meetings today?", which represents the label history 245 for the current utterance 106 of "What about today?". In other examples, the label history 245 comprises all terms preceding the current label of the utterance. For example, for the utterance 106 "play_my_playlist," the label history 245 may correspond to the term "play_my" where the current term (e.g., the next hypothesis term) of the utterance 106 is "playlist." Optionally, the neural oracle search NOS model 320 may receive a language identifier 107 indicating the language of the utterance 106 spoken by the user 104.

[0038] 3A illustrates an example of a rescoring process 300, 300a that includes multiple language-specific neural oracle search NOS models 320S, 320Sa-320Sn, where each language-specific neural oracle search NOS model 320S is trained on pairwise data (i.e., transcribed acoustic training data) of a particular language. Thus, during the second pass 302, the rescoring process 300 selects a language-specific neural oracle search NOS model 320S that corresponds to the language of the utterance 106 from among the multiple language-specific neural oracle search NOS models 320S based on the language identifier 107. Thus, the example rescoring process 300a assumes that the language identifier 107 is made available to select the correct language-specific neural oracle search NOS model 320S.

[0039] Alternatively, FIG. 3B illustrates an exemplary rescoring process 300, 300b comprising a multilingual neural oracle search NOS model 320, 320M. In this example, the multilingual neural oracle search NOS model 320M is trained on pairwise data (i.e., transcribed speech training data) of any number of languages. Thus, the exemplary rescoring process 300b can implement a single multilingual neural oracle search NOS model 320M in a multilingual speech environment. Notably, the exemplary rescoring process 300b does not require the use of any language identifier 107, since the selection of a language-specific neural oracle search NOS model 320S (as described with reference to FIG. 3A) associated with the language of the utterance 106 is not required. Thus, the utterance 106 can comprise a multilingual utterance comprising code-mixing of speech across two or more languages. As used herein, the neural oracle search NOS model 320 may comprise either a language-specific neural oracle search NOS model 320S (FIG. 3A) that the rescoring process 300a selects based on the language identifier 107, or a multilingual neural oracle search NOS model (FIG. 3B).

[0040] 3A and 3B, the neural oracle search NOS model 320 calculates the next label Y given the label history 245. i The neural oracle search NOS model 320 includes a prior model that predicts the prior score of the next label Y based on previously recognized words, word fragments, and / or graphemes. The prior model of the neural oracle search NOS model 320 may include two layers of unidirectional LSTM with 512 units per layer. The prior model is trained by using labeled audio training data and cross-entropy loss. Furthermore, the neural oracle search NOS model 320 includes a posterior model that predicts the posterior score by combining the label history 245 and the sequence of acoustic frames 110 from the first pass 301 in a label-synchronized manner. The posterior model of the neural oracle search NOS model 320 may include two layers of unidirectional LSTM with 512 units per layer and two layers of label-synchronized attention mechanism with 128 units per layer. The posterior model predicts the prior score of the next label Y given the label history 245 and the sequence of acoustic frames 110. i The neural oracle search NOS model 320 is trained with labeled acoustic training data and cross-entropy loss to predict . The neural oracle search NOS model 320 generates an unnormalized likelihood score 325 by summing the token-level prior scores and the token-level posterior scores. Thus, the unnormalized likelihood score 325 is a sequence-level score represented by the sum:

[0041]

number

[0042] In formula 1, S θ1 represents the unnormalized likelihood score 325. The rescorer 350 receives the standalone scores 205, the language model scores 315, and the unnormalized likelihood scores 325 for each of the N candidate hypotheses 204 and generates a respective overall score 355. In particular, the rescorer 350 generates the overall score 355 for each candidate hypothesis 204 based on any combination of the standalone scores 205, the language model scores 315, and the unnormalized likelihood scores 325. In some examples, the rescorer 350 linearly sums the standalone scores 205, the language model scores 315, and the unnormalized likelihood scores 325 to determine a sequence-level overall score 355, as follows:

[0043]

number

[0044] In equation (3), S θ1 represents the unnormalized likelihood score 325. S θ2 represents the external language model score of 315. θ3 represents the single score 205. To optimize the model parameters of the rescorer 350 during training, the rescoring process 300 uses a cross-entropy object between the posterior scores and the sequence-level ground truth distribution. In some examples, the training process assigns the total ground truth distribution to the ground truth transcription while assigning all other candidate hypotheses to zero. In other examples, the training process assigns the full ground truth distribution uniformly across all candidate hypotheses that have a word error rate (WER) less than the best candidate hypothesis (i.e., the ground truth transcription). In yet other examples, the training process applies a softmax function to the negative edit distance between each candidate hypothesis and the ground truth transcription.

[0045] The rescorer 350 then selects the candidate hypothesis 204 from the N candidate hypotheses 204 having the highest overall score 355 as the final transcription 120 of the utterance 106. In the illustrated example, the candidate hypotheses 204 include "PLAY_NEXT_SONG" having an overall score 355 of 0.9, "HEY_NEXT_LONG" having an overall score 355 of 0.3, and "PLAY_NEXT_PONG" having an overall score 355 of 0.5. Continuing with the example, the rescorer 350 selects the candidate hypothesis 204 of "PLAY_NEXT_SONG" (shown by the solid box) having the highest overall score 355 of 0.9 as the transcription 120 (e.g., the final transcription 120b (FIG. 1)). Notably, the candidate hypothesis 204 with the highest individual score 205 (i.e., the likelihood of being the correct transcription) is not the correct candidate hypothesis 204. And, the candidate hypothesis with the highest overall score 355 is the correct transcription from the second pass 302.

[0046] 4 is a flow chart of an exemplary arrangement of operations of a computer-implemented method 400 using a multilingual rescoring model for automatic speech recognition. In operation 402, the method 400 includes receiving a sequence of acoustic frames 110 extracted from speech data corresponding to an utterance 106. In operation 404, during a first pass 301, the method 400 includes processing the sequence of acoustic frames 110 to generate N candidate hypotheses 204, 204a-204n for the utterance 106 using a multilingual speech recognition model (i.e., an automatic speech recognition ASR model) 200. During a second pass 302, for each candidate hypothesis 204 of the N candidate hypotheses 204, the method 400 performs operations 406-412. In operation 406, the method 400 includes using a neural oracle search NOS model 320 to generate a respective unnormalized likelihood score 325. Here, the neural oracle search NOS model 320 generates an unnormalized likelihood score 325 based on the sequence of acoustic frames 110 and the corresponding candidate hypotheses 204. In operation 408, the method 400 includes generating each external language model score 315 using the language model 310. In operation 410, the method 400 includes generating a simplex score 205 that models a priori statistics of the corresponding candidate hypotheses 204 generated during the first pass 301. In operation 412, the method 400 includes generating a total score 355 for each of the candidate hypotheses 255 based on the unnormalized likelihood score 325, the external language model score 315, and the simplex score 205. In operation 414, the method 400 includes selecting the candidate hypothesis 204 having the highest (highest) respective total score 355 from among the N candidate hypotheses 204 as the final transcription 120 of the utterance 106.

[0047] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described herein. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the practice of the inventions described and / or claimed herein.

[0048] Computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and high-speed expansion port 550, and a low-speed interface / controller 560 that connects to low-speed bus 570 and storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be implemented on a common motherboard or in other suitable manners. Processor 510 may process instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memories, as appropriate. Additionally, multiple computing devices 500 may be connected together, with each device providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0049] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-volatile memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) used by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory, read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks and tapes.

[0050] The storage device 530 can provide mass storage for the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be an array of devices including a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, a computer program product is embodied in an information carrier. The computer program product comprises instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0051] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages the non-bandwidth-intensive (low-bandwidth intensive) operations. Such assignment of duties is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 550, and can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 includes various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), and is coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a network device, such as a switch or router, via a network adapter.

[0052] Computing device 500 may be implemented in a number of different forms, as shown, for example as a standard server 500a, or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0053] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be special purpose or general purpose and may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor coupled to receive data and instructions from and / or transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0054] These computer programs (also known as programs, software, software applications or codes) comprise machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0055] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. In general, a computer is operatively coupled to one or more mass storage devices, such as magnetic disks, magneto-optical disks, optical disks, etc., for storing data, to receive data from them, to transfer data to them, or both. However, a computer need not include such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0056] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can be used to provide for user interaction as well. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Input from the user can be received in any form that includes acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0057] A number of embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (400) that, when executed on data processing hardware (510), causes the data processing hardware (510) to perform operations, the operations including: receiving a sequence of extracted acoustic frames (110) from audio data corresponding to an utterance (106); generating N candidate hypotheses (204) for the utterance (106) by processing the sequence of acoustic frames (110) using a multilingual speech recognition model (200) during a first pass (301); and said operations further comprise, during a second pass (302), for each said candidate hypothesis (204) of the N said candidate hypotheses (204): generating respective unnormalized likelihood scores (325) based on the sequence of acoustic frames (110) and the corresponding candidate hypotheses (204) using a neural oracle search NOS model (320); generating a respective external language model score (315) using the external language model (310); generating a simplex score (205) that models the prior statistics of the corresponding candidate hypotheses (204) generated during the first pass (301); generating an overall score (355) for each of the candidate hypotheses (204) based on the unnormalized likelihood scores (325), the external language model scores (315), and the simplex scores (205); and said operation further comprises: selecting, from among the N candidate hypotheses (204), the candidate hypothesis (205) having the highest overall score (355) as a final transcription (120) of the utterance (106). A computer-implemented method (400).

2. Each of the N candidate hypotheses (204) comprises a respective sequence of word or subword labels; Each of the word or subword labels is represented by a respective embedding vector. The computer-implemented method (400) of claim 1.

3. The external language model (310) is trained on text-only data.

3. A computer-implemented method (400) according to claim 1 or 2.

4. The neural oracle search NOS model (320) comprises a language-specific neural oracle search NOS model (320S); A computer implemented method (400) according to any one of claims 1 to 3.

5. The operation further comprises: receiving a language identifier (107) indicating a language of the utterance (106); Selecting a language-specific neural oracle search NOS model (320S) from among a plurality of language-specific neural oracle search NOS models (320S), each of which has been trained for a different language; Equipped with The computer-implemented method (400) of claim 4.

6. The neural oracle search NOS model (320) comprises a multilingual neural oracle search NOS model (320M); A computer implemented method (400) according to any one of claims 1 to 3.

7. The external language model (310) comprises a language-specific external language model (310). A computer implemented method (400) according to any one of the preceding claims.

8. The operation further comprises: receiving a language identifier (107) indicating a language of the utterance (106); selecting a language-specific external language model (310) from among a plurality of language-specific external language models (310), each of which has been trained for a different language; The computer-implemented method (400) of claim 7, comprising:

9. The neural oracle search NOS model (320) comprises two one-way long short-term memory (LSTM) layers. A computer implemented method (400) according to any one of the preceding claims.

10. The multilingual speech recognition model (200) comprises an encoder-decoder architecture comprising a conformer encoder having a plurality of conformer layers and a LSTM decoder having two LSTM layers. A computer implemented method (400) according to any one of the preceding claims.

11. A system (500), comprising: Data processing hardware (510); memory hardware (520) in communication with the data processing hardware (510), the memory hardware (520) storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations; and the operation comprises: receiving a sequence of extracted acoustic frames (110) from audio data corresponding to an utterance (106); generating N candidate hypotheses (204) for the utterance (106) by processing the sequence of acoustic frames (110) using a multilingual speech recognition model (200) during a first pass (301); and said operations further comprise, during a second pass (302), for each said candidate hypothesis (204) of the N said candidate hypotheses (204): generating respective unnormalized likelihood scores (325) based on the sequence of acoustic frames (110) and the corresponding candidate hypotheses (204) using a neural oracle search NOS model (320); generating a respective external language model score (315) using the external language model (310); generating a simplex score (205) that models the prior statistics of the corresponding candidate hypotheses (204) generated during the first pass (301); generating an overall score (355) for each of the candidate hypotheses (204) based on the unnormalized likelihood scores (325), the external language model scores (315), and the simplex scores (205); and said operation further comprises: selecting, from among the N candidate hypotheses (204), the candidate hypothesis (205) having the highest overall score (355) as a final transcription (120) of the utterance (106). System (500).

12. Each of the N candidate hypotheses (204) comprises a respective sequence of word or subword labels; Each of the word or subword labels is represented by a respective embedding vector. The system (500) of claim 11.

13. The external language model (310) is trained on text-only data.

13. The system (500) of claim 11 or 12.

14. The neural oracle search NOS model (320) comprises a language-specific neural oracle search NOS model (320S); A system (500) according to any one of claims 11 to 13.

15. The operation further comprises: receiving a language identifier (107) indicating a language of the utterance (106); selecting a language-specific neural oracle search NOS model (320S) from among a plurality of language-specific neural oracle search NOS models (320S), each of which has been trained for a different language; The system (500) of claim 14, comprising:

16. The neural oracle search NOS model (320) comprises a multilingual neural oracle search NOS model (320M); A system (500) according to any one of claims 11 to 13.

17. The external language model (310) comprises a language-specific external language model (310). A system (500) according to any one of claims 11 to 16.

18. The operation further comprises: receiving a language identifier (107) indicating a language of the utterance (106); selecting a language-specific external language model (310) from among a plurality of language-specific external language models (310), each of which has been trained for a different language; Equipped with 20. The system (500) of claim 17.

19. The neural oracle search NOS model (320) comprises two one-way long short-term memory (LSTM) layers. A system (500) according to any one of claims 11 to 18.

20. The multilingual speech recognition model (200) a conformer encoder having a plurality of conformer layers; an LSTM decoder having two LSTM layers; an encoder-decoder architecture comprising: A system (500) according to any one of claims 11 to 19.

Images