Contextual spelling correction (CSC) for automatic speech recognition (ASR)

Abstract

Novel solutions for speech recognition provide context spelling correction (CSC) for automatic speech recognition (ASR). Disclosed examples include receiving an audio stream; performing ASR processing on the audio stream to produce ASR hypotheses; receiving a context list; and performing spelling correction based at least on the ASR hypotheses and the context list to produce an output text sequence. Using a context spelling correction (CSC) model on top of an ASR model, this eliminates the need to change the original ASR model. This permits runtime user customization based on context data, even for large context lists. Some examples include filtering ASR hypotheses for an audio stream, and determining whether to trigger spelling correction for an ASR hypothesis based at least on ASR hypothesis filtering. Some examples include using pre-processed transcriptions with context phrases to generate text-to-speech (TTS) audio to train a CSC model.

Description

BACKGROUND

[0001] Automatic speech recognition (ASR) is used for purposes such as input to a digital assistant to, for example, initiate a phone call, compose a message, and manage calendar events. However, such purposes often require matching ASR results to context-specific words, such as contact list names. Unfortunately, some contact list names have unique spellings that can not match ASR results, resulting in failed attempts. Other specialized languages, such as obscure medical and other industry-specific terminology, can also increase the word error rate (WER), resulting in mispellings for transcriptions.

[0002] Existing solutions, such as context language models (LMs) that utilize bias finite state machines (FSTs) to provide on-the-fly rescoring and bias encoders that require customized training, often suffer from performance degradation issues. The degraded performance can manifest as higher latency for long context lists, e.g., lists of context-specific words such as contact list names and specialized terminology. A challenge in conjunction with existing solutions is that, for certain scenarios, the dynamic nature of many context lists prevents context-specific words from being available during ASR training. SUMMARY

[0003] The disclosed examples are described in detail below with reference to the accompanying drawings listed after the detailed description. The following summary is provided to illustrate some of the examples disclosed herein. This summary does not mean to limit any examples to any particular configuration or order of operation.

[0004] A solution for speech recognition provides context spelling correction (CSC) for automatic speech recognition (ASR). Disclosed examples include receiving an audio stream, performing ASR processing on the audio stream to produce ASR hypotheses, receiving a context list, and performing spelling correction based at least on the ASR hypotheses and the context list to produce an output text sequence. Using a context spelling correction (CSC) model on top of an ASR model excludes the need to change the original ASR model. This permits runtime user customization based on context data, even for large context lists. Some examples include filtering ASR hypotheses for an audio stream and determining whether to trigger spelling correction for the ASR hypotheses based at least on the ASR hypothesis filtering. Some examples include using preprocessed transcriptions with context phrases to generate text-to-speech (TTS) audio to train a CSC model. BRIEF DESCRIPTION OF DRAWINGS

[0005] The disclosed examples are described in detail below with reference to the accompanying drawings listed after the detailed description:

[0006] Figure 1Arrangements for speech recognition that advantageously employ contextual spelling correction (CSC) for automatic speech recognition (ASR) are illustrated;

[0007] Figure 2 Further details of the CSC model illustrated in Figure 1 are illustrated in further detail;

[0008] Figure 3 Further details of the CSC model of Figure 2 are illustrated in further detail;

[0009] Figure 4 Further details of the CSC model of Figure 2 are also illustrated in further detail;

[0010] Figure 5 Further details of the CSC model of Figure 2 are also illustrated in further detail;

[0011] Figure 6 is a flow diagram illustrating example operations associated with the arrangements of Figure 1 is a flow diagram illustrating example operations associated with the arrangements of

[0012] Figure 7 is a process flow diagram illustrating example operations and data involved in training the CSC model of Figure 2 is a process flow diagram illustrating example operations and data involved in training the CSC model of

[0013] Figure 8 is another flow diagram illustrating example operations associated with the arrangements of Figure 1 is another flow diagram illustrating example operations associated with the arrangements of

[0014] Figure 9 is a block diagram of an example computing environment suitable for implementing some of the various examples disclosed herein.

[0015] Corresponding reference numerals indicate corresponding parts throughout the several figures. DETAILED DESCRIPTION

[0016] Various examples will be described in detail with reference to the drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made to particular examples and implementations throughout this disclosure are for illustrative purposes only and are not intended to limit all examples unless explicitly indicated otherwise.

[0017] A novel solution for speech recognition provides context spelling correction (CSC) for automatic speech recognition (ASR). Disclosed examples include receiving an audio stream; performing ASR processing on the audio stream to produce ASR hypotheses; receiving a context list; and performing spelling correction based at least on the ASR hypotheses and the context list to produce an output text sequence. A context spelling correction (CSC) model is used on top of an ASR model, which precludes the need to change the original ASR model. This permits runtime user customization based on context data, even for large context lists. Some examples include filtering ASR hypotheses for an audio stream, and determining whether to trigger spelling correction for the ASR hypotheses based at least on the ASR hypothesis filtering. Some examples include using pre-processed transcriptions with context phrases to generate text-to-speech (TTS) audio to train a CSC model.

[0018] Aspects of the present disclosure improve the speed and accuracy of speech recognition by receiving a context list and performing spelling correction based at least on the ASR hypotheses and the context list to produce an output text sequence. This approach avoids latency that occurs when using biasing ASR encoders with long context lists, and also avoids performance issues associated with context language model (LM) solutions that change the ASR decoding process.

[0019] The disclosed CSC model corrects context-dependent recognition errors in transducer-based ASR systems. Context information is incorporated into a spelling correction model with a shared context encoder, and filtering is used to handle large context lists. In some examples, the word error rate (WER) can be reduced by about half, even for out-of-vocabulary terms (e.g., personal names) that were not seen during training. By using an autonomous correction model that does not change the original transducer model structure, the baseline ASR model is not at risk of performance degradation. Another benefit is that the approach can be applied to different domains by changing the CSC model without retraining the original ASR model. Furthermore, the CSC model can be lightweight, simplifying deployment and facilitating operation in resource-constrained environments.

[0020] In some examples, each audio utterance produces multiple speech recognition hypotheses, and the top K are selected for possible post-processing (spelling correction). A hypothesis filter determines, based on (at least) the speech recognition hypotheses, whether to perform post-processing. If so, the speech recognition hypotheses and an initial context list are passed through a context list filter to obtain a pre-selected context list. The speech recognition hypotheses and the pre-selected context list are passed through a CSC model to obtain a final result. Otherwise, post-processing is skipped.

[0021] Figure 1An arrangement 100 for speech recognition that advantageously employs CSC for ASR is illustrated. An audio stream 102 is received (captured) from a speaker 106 by a microphone 104 for ASR and produces an output text sequence 170 that is passed to a digital assistant 180 or a transcription service 190. The output text sequence 170 is advantageously subjected to CSC so that the output text sequence 170 correctly matches the spelling of even obscure words in a context list 160.

[0022] The audio stream 102 is received and segmented into multiple audio segments 110 by an audio segmenter 108. As illustrated, the multiple audio segments 110 include an audio segment 111, an audio segment 112, and an audio segment 113. The audio segment 111 is "What is my meeting with Aliza Friedman?" This will be interpreted by the digital assistant 180 as a query to a calendar function 183. The name "Aliza Friedman" is illustrated in bold to emphasize because in this example, "Aliza Friedman" is mis-spelled by the ASR model 120 as "Alyssa Friedman." If allowed to persist, this mis-spelling can result in an incorrect answer from the digital assistant 180 because the calendar function 183 will search for events that list "Alyssa Friedman" as a participant, instead of searching for events that list "Aliza Friedman" as a participant.

[0023] The ASR model 120 is illustrated as having an encoder and a decoder that can each include a neural network (NN). In some examples, the ASR model 120 includes a recurrent neural network transducer (RNN-T) that performs end-to-end (E2E) ASR. The ASR model 120 outputs text sequences as ASR hypotheses 130, which are illustrated as including an ASR hypothesis 131, an ASR hypothesis 132, and an ASR hypothesis 131. In some examples, multiple ASR hypotheses are generated for a single utterance (segmented into one of the audio segments 110). In the illustrated example, the ASR hypotheses 131-133 all correspond to the audio segment 111. The ASR hypothesis 131 is "What is my meeting with Alyssa Friedman (mis-spelling of "Aliza")." The ASR hypothesis 132 is "What is my meeting with Aliza Friendman (mis-spelling of "Friedman")." The ASR hypothesis 133 is "What is my meeting with Alysa Friend man (mis-spellings of both "Aliza" and "Friedman")."

[0024] Each of the ASR hypotheses 130 is scored. For example, ASR hypothesis 131 ("What is my meeting with Alyssa Friedman") has a score of -0.1, ASR hypothesis 132 ("What is my meeting with Aliza Friendman") has a score of -0.2, and ASR hypothesis ("What is my meeting with Alysa Friendman") has a score of -0.3. Subsequently, the score vector is [-0.1, -0.2, -0.3], which achieves a ranking of the ASR hypotheses 131-133.

[0025] To address potential issues with general domain regression for large context lists, the top-ranked ASR hypotheses 130 are passed to a two-stage filter 140. For example, if only the top two pass, this can include ASR hypothesis 131 and ASR hypothesis 132, but not ASR hypothesis 133. The filter 140 includes a domain classifier 142 that acts as an ASR hypothesis filter and determines whether to trigger spelling correction (e.g., CSC) for any of the ASR hypotheses. If either ASR hypothesis 131 or ASR hypothesis 132 triggers spelling correction, then both ASR hypothesis 131 and ASR hypothesis 132 will be sent to a context filter 144 and subsequently to a CSC model 200 for spelling correction. If neither ASR hypothesis 131 nor ASR hypothesis 132 triggers spelling correction, then the context filter 144 and the CSC model 200 are skipped.

[0026] The context filter 144 performs context pre-selection and includes a relevance ranker 146 that receives a relatively large initial context list 150 and a preference ranker 148 that introduces context ranking weights 156. In some examples, the initial context list 150 includes a list of personal names, a list of location names (e.g., street names and city names that can have uncommon spellings), and / or a list of professional terms 154 (e.g., medical, legal, financial, or other terms). The context filter 144 considers similarities between ASR hypothesis 131 and / or ASR hypothesis 132 and items in the initial context list 150 (based on and preference information (as determined by the preference ranker 148)).

[0027] The relevance ranker 146 includes an edit distance filter and is used to constrain context enumeration according to edit distance between the initial context list 150 and the ASR input (ASR hypothesis 131 or ASR hypothesis 132) in order to speed up model decoding. The edit distance filter is described as follows:

[0028]

[0029] where si is a segment truncated from the input text having the same length as a certain context phrase x j starting from the i-th word, and is a relevance ranking weight of the j-th context phrase. In some scenarios, the context phrase hidden state representation of a certain user (e.g., speaker 106) can be generated in advance to reduce inference cost.

[0030] The context ranking weight 156 reflects, for example, user preferences indicated by the frequency of use of a particular context list item (e.g., a contact name). In some examples, the context ranking weight 156 is used together with an edit distance filter weight to pre-select contexts, e.g., to narrow down the initial context list 150 to a pre-selected context list 160. The final (pre-selected) context list 160 from the preference ranker 148 is selected according to:

[0031] c = topN(k * w r + w p ) Equation (2)

[0032] where c is the list of selected context phrases, w r is the relevance ranking weight, and w p is the preference ranking weight, and k is a weight used to adjust the contribution of the two weights. In some examples, k is set to 0.5. This narrowing down of the initial context list 150 to the context list 160 occurs for each set of ASR hypotheses selected for spelling correction.

[0033] The top ASR hypotheses, i.e., ASR hypothesis 131 and ASR hypothesis 132, are passed to the CSC model 200, which is described and explained in further detail with respect to Figures 2-5 The CSC model 200 receives at least a portion of the initial context list 150 as the (pre-selected) context list 160, e.g., the contact name 161 of “Aliza Friedman” possibly along with other similar contact names such as contact name 162 and contact name 163.

[0034] As described below, the CSC model 200 corrects the spelling of “Alyssa Friedman” to “Aliza Friedman” in the corrected ASR hypothesis 131a and outputs it as the output text sequence 170. The final decoding result is obtained by ranking the ASR hypotheses:

[0035] H * = argmax(λ SR log P i + λ CSC log Pij ) Equation (3)

[0036] where λ SR and λ CSC are weights for the ASR and CSC scores. In some examples, the set of CSC hypotheses {Hii, H2,..., HN} is generated by a beam search mechanism.

[0037] Depending on the particular ASR task, the output text sequence 170 is provided to a digital assistant 180 and / or a transcription service 190. The digital assistant 180 is configured to perform various actions with respect to the output text sequence 170, such as making a phone call, generating a message, and performing a calendar operation, using a phone function 181, a messaging function 182, a calendar function 183, and / or another function. The transcription service 190 generates a transcription 192 of the audio stream 102.

[0038] Turning now to Figure 2 the CSC model 200 is described in further detail. The CSC model 200 is a sequence-to-sequence (seq2seq) model with an encoder 300 and a decoder 330. The encoder 300 includes a text encoder 310 and a context encoder 320. The text encoder 310 takes as input the ASR hypotheses 131, while the context encoder 320 takes as input the context list 16. The context encoder 320 extracts context phrase embeddings. The decoder 330 attends to both the encoders 310 and 320 to obtain information from the ASR hypotheses 131 and the context list 160 to correct context misspellings. As indicated, the attention network hypothesis 202 and the attention network context 204 are provided to the decoder 330. The text encoder 310 and the context encoder 320 share parameters. To consider context information during spelling correction, the context encoder 320 encodes context phrases into hidden embeddings. In some examples, teacher-student learning and quantization are used in order to provide a lightweight model (e.g., relatively fast and small).

[0039] Figure 3 Further details of the text encoder 310, the context encoder 320, and the decoder 330 are illustrated. In some examples, the components are transformer-based, and the parameters of the two encoders (the text encoder 310 and the context encoder 320) are shared. The ASR hypotheses 131 provide embeddings 316 to the text encoder 310 (which can have N instances). The text encoder 310 can include one or more NNs, and is illustrated as having a self-attention network 312 and a feed-forward network 314.

[0040] The context list 160 provides an embedding 326 for the context encoder 320. The context encoder 320 can include one or more NNs, and is illustrated as having a self-attention network 322 and a feed-forward network 324. The output of the context encoder 320 provides a context encoder hidden state 328, which is a hidden state (representation) of the context encoder 320. The context encoder hidden state 328 and the output of the text encoder 310 are provided to a speech recognition context attention network 336 within the decoder 330. Shared parameters for the text encoder 310 and the context encoder 320 make this arrangement equivalent to using a single encoder network. In some examples, using a single encoder is feasible because ASR assumes that both the text and the context phrase are transcriptions that can be processed by the same network. In some scenarios, such as domains with personal names, the training context list (e.g., the training context list 722) is not large enough to cover all possible word tokens or patterns, so using the same network can enable the context encoder 320 and the text encoder 310 to benefit from each other. Additionally, a single encoder network can make the CSC model 200 smaller. Figure 7

[0041] The decoder 330, which can also have N instances, can include one or more NNs, and is illustrated as also having a self-attention network 332 and a feed-forward network 334. The decoder 330 outputs an output probability 340, which is right-shifted and returned as feedback output 342.

[0042] Figure 4 A block encoder 400 is illustrated that can be used as the text encoder 310 and / or the context encoder 320. The input is fed to a normalization stage 402, and then to a self-attention network 404 (e.g., the self-attention network 312 or 322). At each decoding step, the query vector Q also attends to the user’s context phrase embedding. This attention is added to the attention of the text encoder 310 to produce the final attention during decoding. The key vector K and the value vector V can also be used. Knowledge distillation and quantization are also employed to further reduce the model size and improve inference efficiency. The student model has the same structure as the teacher model but with smaller hidden state dimensions.

[0043] The output of the self-attention network 404 is added to the input and fed to another normalization stage 408, and then to a feed-forward network 410 (e.g., the feed-forward network 314 or 324). The output of the feed-forward network 410 is then added to the input of the normalization stage 408.

[0044] Figure 5 ​A block decoder 500 that can be used as the decoder 330 is illustrated. Input is fed to a normalization stage 502 and then to a self-attention network 504 (e.g., the self-attention network 332). The output of the self-attention network 504 is added to the input and fed to another normalization stage 508 and then to an encoder-decoder attention network 510 (e.g., the attention network 336). The output of the encoder-decoder attention network 510 is then added to the input of the normalization stage 508 and fed to another normalization stage 514. The output of the normalization stage 514 is then fed to a feed-forward network 516 (e.g., the feed-forward network 334). The output of the feed-forward network 516 is then added to the input of the normalization stage 514.

[0045] Figure 6 is a flowchart 600 illustrating example operations involved in performing speech recognition. In some examples, the operations described for the flowchart 600 are performed by the computing device 900 of Figure 9 . The flowchart 600 begins with operation 602, which includes training the CSC model 200 (contextual spell checker) as described by the process flow 700 of Figure 7 . Operation 604 includes receiving the audio stream 102, and operation 606 includes segmenting the audio stream 102 into a plurality of audio segments 110. Operation 608 includes performing ASR processing on the audio stream 102 to generate ASR hypotheses 131, which in some examples include a sequence of text. In some examples, performing the ASR process includes performing ASR with a NN. In some examples, the ASR hypotheses 131 include hypotheses for speech in an audio segment 111 of the plurality of audio segments 110.

[0046] Operation 610 performs domain classification, which filters the ASR hypotheses 130 (including the ASR hypotheses 131 and the ASR hypotheses 132). Decision operation 612 uses the results of the filtering operation 610 to determine whether to trigger spelling correction for the ASR hypotheses 131 based on the ASR hypotheses filtering of operation 610. If spelling correction is not triggered, the flowchart 600 skips the CSC and jumps to operation 624, in which the ASR hypotheses 131 (the highest ranked ASR hypothesis) is output as the output text sequence 170. This does not mean that no other spelling correction or word replacement is used (e.g., the transcription 192 can be subject to other spelling check or automatic editing processes), only that the ASR hypotheses 131 that is applied is not passed through the CSC model 200. Operation 624 is described in further detail below.

[0047] If spelling correction is triggered, the CSC model 200 receives the ASR hypotheses 131, 132 and the context list 160 in operation 614. In some examples, the context list 160 includes a plurality of text sequences, e.g., a list of contact names 161-163, a list of location names, and / or a dictionary of professional terms. Operation 616 includes performing context filtering, and is accomplished at least in part using operations 618 and 620. Operation 618 ranks the ASR hypotheses 131 and 132 using the relevance ranker 146, and operation 620 ranks the preferences using the preference ranker 148.

[0048] Operation 622 includes performing spelling correction to produce the output text sequence 170 based at least on determining that spelling correction is to be triggered for the ASR hypothesis 131 (and the ASR hypothesis 132). In some examples, the spelling correction includes the CSC. In some examples, performing the spelling correction includes performing the spelling correction with the NN (e.g., within the CSC model 200). In some examples, performing the spelling correction includes inputting the ASR hypothesis 131 into the text encoder 310 and / or inputting the context list 160 into the context encoder 320. As part of operation 622, the context encoder 320 extracts the context phrase embeddings. In some examples, performing the spelling correction includes passing the output of the text encoder 310 and the output of the context encoder 320 to the decoder 330.

[0049] In operation 624, the digital assistant 180 and / or the transcription service 190 receives the corrected ASR hypothesis 131a as the output text sequence 170, or the highest ranked ASR hypothesis 131 if spelling correction was not triggered in decision operation 612. Operation 626 includes performing an action with the output text sequence 170. In some examples, the action is selected from a list including: generating a transcription 192 of the audio stream 102, initiating a phone call to a contact identified in the audio stream 102, generating a message to a contact identified in the audio stream 102, and responding to a query within the audio stream 102.

[0050] Figure 7 is a diagram of a process flow 700 illustrating example operations and data involved in training the CSC model 200. In some examples, the operations described with respect to the flowchart 700 are performed by the CSC model 200, the digital assistant 180, and / or the transcription service 190. Figure 9computing device 900 executes. Process flow 700 begins with operation 702, which generates training data, e.g., training scripts 720 and training context list 722. Training scripts 720 are preprocessed transcriptions with context phrases built by combining sentence patterns with name tags (or other context phrases), such as “call <person name>” and “do I have an email from <person name>?” In some examples, the contents of training context list 722 are randomly selected from a source of context phrases.

[0051] Text-to-speech (TTS) operation 704 generates TTS audio (training audio stream 724) from training scripts 720. Operation 706 performs ASR to generate ASR hypotheses 726 with error patterns. ASR hypotheses 726 are input into CSC training operation 708, which trains untrained CSC model 200a using training context list 722 as ground truth. This results in a trained version of CSC model 200.

[0052] In some instances, after teacher model training, knowledge distillation is also employed to further reduce model size and improve inference efficiency. This enables the use of CSC model 200 on devices with strict computational resource constraints. In some examples, the loss function for the student model is:

[0053] L = aL soft + (1 - a)L hard Equation (4)

[0054] L hard = cross_entropy(y S , y) Equation (5)

[0055]

[0056] where L hard is the cross-entropy loss of student model output y S and reference y, L soft is the KL divergence of student model output y S and teacher model output y T , T is a temperature parameter, and a is a weight value.

[0057] Figure 8 is a flowchart 800 illustrating example operations involved in performing speech recognition. In some examples, the operations described for flowchart 800 are performed by Figure 9by the computing device 900. The flowchart 800 begins with operation 802, which includes receiving an audio stream. Operation 804 includes performing ASR processing on the audio stream to produce an ASR hypothesis. Operation 806 includes receiving a context list. Operation 808 includes performing spelling correction based at least on the ASR hypothesis and the context list to produce an output text sequence.

[0058] Additional Examples

[0059] An example method of speech recognition includes receiving an audio stream; performing automatic speech recognition (ASR) processing on the audio stream to produce an ASR hypothesis; receiving a context list; performing spelling correction based at least on the ASR hypothesis and the context list to produce an output text sequence.

[0060] An example system for speech recognition includes a processor; and a computer- readable medium storing instructions that are operable, when executed by the processor, to: receive an audio stream; perform automatic speech recognition (ASR) processing on the audio stream to produce an ASR hypothesis; receive a context list; perform spelling correction based at least on the ASR hypothesis and the context list to produce an output text sequence.

[0061] One or more example computer storage devices having stored thereon computer- executable instructions that, when executed by a computer, cause the computer to perform operations including: receiving an audio stream; performing automatic speech recognition (ASR) processing on the audio stream to produce an ASR hypothesis; receiving a context list; performing spelling correction based at least on the ASR hypothesis and the context list to produce an output text sequence.

[0062] As alternatives or supplements to other examples described herein, examples can include any combination of the following:

[0063] - the spelling correction includes context spelling correction (CSC);

[0064] - performing an action with the output text sequence, wherein the action is selected from a list including: generating a transcription of the audio stream, initiating a phone call to a contact identified in the audio stream, generating a message to a contact identified in the audio stream, and responding to a query within the audio stream;

[0065] - performing spelling correction includes inputting the ASR hypothesis into a text encoder;

[0066] - performing spelling correction includes inputting the context list into a context encoder;

[0067] - performing spelling correction includes passing an output of the text encoder and an output of the context encoder to a decoder;

[0068] filtering ASR hypotheses for the audio stream;

[0069] determining whether to trigger spelling correction for the ASR hypothesis based at least on the ASR hypothesis filtering;

[0070] performing spelling correction to produce the output text sequence includes performing spelling correction to produce the output text sequence based at least on determining to trigger spelling correction for the ASR hypothesis;

[0071] outputting the ASR hypothesis as the output text sequence based at least on determining not to trigger spelling correction for the ASR hypothesis;

[0072] training the contextual spelling checker, wherein the training includes generating TTS audio using pre-processed transcriptions with contextual phrases;

[0073] the ASR hypothesis includes a text sequence;

[0074] the contextual list includes a text sequence;

[0075] the contextual list includes contact names in a contact list;

[0076] the contextual list includes location names;

[0077] the contextual list includes a dictionary of professional terms;

[0078] segmenting the audio stream into a plurality of audio segments, wherein the ASR hypothesis includes a hypothesis of speech in an audio segment of the plurality of audio segments;

[0079] receiving the output text sequence by a digital assistant;

[0080] performing the ASR process includes performing ASR with a NN;

[0081] performing spelling correction includes performing spelling correction with a NN;

[0082] performing contextual filtering;

[0083] performing contextual filtering includes ranking relevancy and preference; and

[0084] extracting contextual phrase embeddings.

[0085] While aspects of the disclosure have been described in accordance with various examples and their associated operations, it should be understood that the operations of the examples can be combined, other examples can be utilized, and the operations can be implemented in a different order than described.

[0086] Example Operating Environment

[0087] Figure 9 is a block diagram of an example computing device 900 for implementing aspects disclosed herein, and is designated generally as computing device 900. Computing device 900 is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the examples disclosed herein. Neither should computing device 900 be interpreted as having any dependency or requirement relating to any one or combination of components / modules illustrated. The examples disclosed herein can be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program components, being executed by the computers or other machines such as personal data assistants or other handheld devices. Generally, program components including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks or implements particular abstract data types. The disclosed examples can be practiced in a variety of system configurations, including personal computers, laptop computers, smart phones, mobile tablets, hand-held devices, consumer electronics, professional computing devices, and the like. The disclosed examples can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0088] Computing device 900 includes bus 910 which directly or indirectly couples the following devices: computer storage memory 912, one or more processors 914, one or more presentation components 916, I / O ports 918, I / O components 920, power supply 922, and network components 924. While the computing device 900 is depicted with what appears to be a single bus 910, multiple buses can be present and used to interconnect the depicted devices. For example, different busses can be used to interconnect the various components discussed herein and a storage area network can be present, with its own network architecture.

[0089] Bus 910 represents what can be one or more busses, such as an address bus, a data bus, or a combination thereof. Various buses can be present in computing device 900 and can be used for information storage in memory 912, to receive processor instructions, to perform the tasks described herein, and the like. Additionally, different buses can interconnect the different components within the depicted environments. Figure 9 The various components described can be represented with different means, for example, a presentation component such as a display device is an I / O component and some examples of processors have their own memory. There is no distinction between the classifications such as “workstation,” “server,” “laptop,” “handheld device,” etc., as all are considered to be within the scope of the examples described herein. Figure 9Within the scope of this document and referred to herein as a "computing device". Memory 912 may take the form of a computer storage medium reference and is operable to provide storage for computer-readable instructions, data structures, program modules, and other data for computing device 900. In some examples, memory 912 stores one or more of an operating system, a general-purpose application platform, or other program modules and program data. Thus, memory 912 is capable of storing and accessing data 912a and instructions 912b, which can be executed by processor 914 and configured to perform the various operations disclosed herein.

[0090] In some examples, memory 912 includes computer storage media in the form of volatile and / or non-volatile memory, removable or non-removable memory, data disks in a virtual environment, or combinations thereof. Memory 912 may include any number of memories associated with or accessible to computing device 900. Memory 912 may be internal to computing device 900 (e.g., Figure 9 The memory 912 may be located outside the computing device 900 (not shown), or both (not shown). Examples of memory 912 include, but are not limited to, random access memory (RAM); read-only memory (ROM); electronically erasable programmable read-only memory (EEPROM); flash memory or other memory technologies; CD-ROM, digital versatile disc (DVD) or other optical or holographic media; magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices; memory connected to an analog computing device; or any other medium used to encode desired information and accessed by the computing device 900. Additionally or alternatively, memory 912 may be distributed across multiple computing devices 900, for example, in a virtualized environment in which instruction processing is performed on multiple devices 900. For the purposes of this disclosure, “computer storage medium,” “computer storage memory,” “memory,” and “memory device” are synonymous terms for computer storage memory 912, and none of these terms includes a carrier wave or propagation signaling.

[0091] The processor 914 can include any number of processing units from various entities such as the memory 912 or the I / O components 920. In particular, the processor 914 is programmed to execute computer-executable instructions for implementing aspects of the present disclosure. These instructions can be executed by the processor, by multiple processors within the computing device 900, or by processors external to the client computing device 900. In some examples, the processor 914 is programmed to execute instructions such as those shown in the flowcharts discussed below and depicted in the accompanying drawings. Also, in some examples, the processor 914 represents one implementation of an analog technique that performs the operations described herein. For example, these operations can be performed by an analog client computing device 900 and / or a digital client computing device 900. The presentation component 916 presents data indications to a user or other devices. Example presentation components include a display device, a speaker, a printing component, a vibrating component, and the like. Those skilled in the art will appreciate and understand that computer data can be presented in a variety of ways, such as visually in a graphical user interface (GUI), aurally through a speaker, wirelessly between computing devices 900, through a wired connection, or otherwise. The I / O ports 918 allow the computing device 900 to be logically coupled to other devices including the I / O components 920, some of which can be built in. Example I / O components 920 include, for example and without limitation, a microphone, a joystick, a game pad, a satellite antenna, a scanner, a printer, a wireless device, and the like.

[0092] The computing device 900 can operate in a network environment using logical connections to one or more remote computers. In some examples, the network components 924 include network interface cards and / or computer-executable instructions for operating network interface cards (e.g., drivers). Communication between the computing device 900 and other devices can occur using any protocol or mechanism over any wired or wireless connection. In some examples, the network components 924 are operable to communicate data wirelessly between public, private, or hybrid (public and private) devices using short-range communication technologies (e.g., near-field communication (NFC), Bluetooth®, Bluetooth® Low Energy, Wi-Fi®, Wi-Fi® Direct, ZigBee®, Z-Wave®, Thread®, and the like) or combinations thereof using transmission protocols between devices. The network components 924 communicate with cloud resources 928 across a network 930 through a wireless communication link 926 and / or a wired communication link 926a. Various different examples of communication links 926 and 926a include wireless connections, wired connections, and / or dedicated links, and in some examples, at least a portion is routed through the Internet. TM Brand communications, and the like) or combinations thereof. The network components 924 communicate with cloud resources 928 across a network 930 through a wireless communication link 926 and / or a wired communication link 926a. Various different examples of communication links 926 and 926a include wireless connections, wired connections, and / or dedicated links, and in some examples, at least a portion is routed through the Internet.

[0093] Although described in conjunction with an example computing device 900, the various examples of this disclosure can be implemented using a multitude of other general-purpose or special-purpose computing system environments, configurations, or devices. Examples of well-known computing systems, environments, and / or configurations applicable to various aspects of this disclosure include, but are not limited to: smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or communication devices with wearable or accessory form factors (e.g., watches, glasses, headphones, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including the above-mentioned systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, holographic devices, and so on. Such systems or devices can accept input from users in any manner, including from input devices such as keyboards or pointing devices, gesture input, proximity input (such as by hovering), and / or voice input.

[0094] The examples of this disclosure can be described in the general context of computer-executable instructions (such as program modules) that are executed by one or more computers or other devices, including software, firmware, hardware, or combinations thereof. Computer-executable instructions can be organized into one or more computer-executable components or modules. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform a particular task or implement a particular abstract data type. Any number of such components or modules and any organization thereof can be used to implement aspects of this disclosure. For example, aspects of this disclosure are not limited to the specific computer-executable instructions or specific components or modules illustrated in the accompanying drawings and described herein. Other examples of this disclosure may include different computer-executable instructions or components having more or fewer functions than those shown and described herein. In examples involving general-purpose computers, aspects of this disclosure transform a general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

[0095] By way of example, and not limitation, computer-readable media include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media are tangible and non-transitory and exclude carrier waves and communication signals. Computer storage media, as used herein, are hardware-based and exclude signals per se. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, solid state drives, flash memory, phase change memory (“PCM”), other types of random access memory (“RAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), other types of random access memory (“RAM”), read only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technology, compact disc read-only memory (“CD-ROM”), digital versatile disks (including the Blu-ray and DVD+RW Discs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store information and that can be accessed by a computing device. By contrast, communication media

[0096] The order of execution or performance of the operations in examples of the disclosure illustrated and described herein need not be limited to the order described. For example, it is contemplated that where discussed operations can be performed in an order other than that described, in another order that is different than that described, or in various orders or concurrently. When introducing elements of aspects of the disclosure or the examples thereof, the articles "a," "an," "the" and "said" are intended to mean that there are one or more of the elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there can be additional elements other than the listed elements. The term "exemplary" is intended to mean "an example of." The phrase "one or more of the following: A, B, and C" means "at least one of A and / or at least one of B and / or at least one of C."

[0097] Having described aspects of the disclosure in detail, it will be apparent that modifications and variations are possible without departing from the scope of aspects of the disclosure defined by the appended claims. It is therefore intended that within the scope of aspects of the disclosure, there is appended claims, that alterations and modifications of the disclosure will occur to others. It is intended that the above description be considered as illustrative only of the aspects of the disclosure and not in a limiting sense, as it is understood that various modifications and changes can be made by those skilled in the art without departing from the scope of aspects of the disclosure.

Claims

1. A speech recognition method, the method comprising: Receive audio stream; Automatic speech recognition (ASR) processing is performed on the audio stream to generate ASR hypotheses; Receive the initial context list; Perform context filtering, wherein the context filtering includes narrowing the initial context list to a pre-selected context list based on the user's preferences; as well as Spell correction is performed at least based on the ASR assumptions and the context list to produce an output text sequence, wherein performing spell correction includes: The ASR hypothesis is input into the text encoder; Input the pre-selected context list into the context encoder; and The output of the text encoder and the output of the context encoder are passed to the decoder.

2. The method of claim 1, further comprising: An action is performed using the output text sequence, wherein the action is selected from a list including: The system generates a transcription of the audio stream, initiates a phone call with a contact identified in the audio stream, generates a message for the contact identified in the audio stream, and responds to queries within the audio stream.

3. The method of claim 1, further comprising: Filter ASR assumptions for the audio stream; Whether to trigger spell correction based on the ASR hypothesis is determined at least based on the ASR hypothesis filtering. Performing spell correction to generate the output text sequence includes: The spelling correction is performed to produce the output text sequence based at least on the determination that spelling correction against the ASR hypothesis should be triggered; as well as The ASR hypothesis is output as the output text sequence, based at least on the premise that spelling corrections for the ASR hypothesis are not triggered.

4. The method of claim 1, further comprising: Training a contextual spell checker, wherein the training includes generating text-to-speech (TTS) audio using preprocessed transcription with contextual phrases.

5. A system for speech recognition, the system comprising: processor; as well as A computer-readable medium storing instructions operable, when executed by the processor, for: Receive audio stream; Automatic speech recognition (ASR) processing is performed on the audio stream to generate ASR hypotheses; Receive the initial context list; Perform context filtering, wherein the context filtering includes narrowing the initial context list to a pre-selected context list based on the user's preferences; as well as Spell correction is performed at least based on the ASR assumptions and the context list to produce an output text sequence, wherein performing spell correction includes: The ASR hypothesis is input into the text encoder; Input the pre-selected context list into the context encoder; and The output of the text encoder and the output of the context encoder are passed to the decoder.

6. The system of claim 5, wherein the instructions are further operable to: An action is performed using the output text sequence, wherein the action is selected from a list including: The system generates a transcription of the audio stream, initiates a phone call with a contact identified in the audio stream, generates a message for the contact identified in the audio stream, and responds to queries within the audio stream.

7. The system of claim 5, wherein the instructions are further operable to: Filter ASR assumptions for the audio stream; Whether to trigger spell correction based on the ASR hypothesis is determined at least based on the ASR hypothesis filtering. Performing spell correction to generate the output text sequence includes: The spelling correction is performed to produce the output text sequence based at least on the determination that spelling correction against the ASR hypothesis should be triggered; as well as The ASR hypothesis is output as the output text sequence, based at least on the premise that spelling corrections for the ASR hypothesis are not triggered.

8. The system of claim 5, wherein the instructions are further operable to: Training a contextual spell checker, wherein the training includes generating text-to-speech (TTS) audio using preprocessed transcription with contextual phrases.

9. One or more computer storage devices storing computer-executable instructions that, when executed by a computer, cause the computer to perform various operations, including: Receive audio stream; Automatic speech recognition (ASR) processing is performed on the audio stream to generate ASR hypotheses; Receive the initial context list; Perform context filtering, wherein the context filtering includes narrowing the initial context list to a pre-selected context list based on the user's preferences; as well as Spell correction is performed at least based on the ASR assumptions and the context list to produce an output text sequence, wherein performing the spell correction operation includes: The ASR hypothesis is input into the text encoder; Input the pre-selected context list into the context encoder; and The output of the text encoder and the output of the context encoder are passed to the decoder.

10. The computer storage device of claim 9, wherein the operation further comprises: An operation is performed using the output text sequence, wherein the operation is selected from a list including the following: The system generates a transcription of the audio stream, initiates a phone call with a contact identified in the audio stream, generates a message for the contact identified in the audio stream, and responds to queries within the audio stream.

11. The computer storage device of claim 9, wherein the operation further comprises: Filter ASR assumptions for the audio stream; Whether to trigger spell correction based on the ASR hypothesis is determined at least based on the ASR hypothesis filtering. Performing spell correction to generate the output text sequence includes: The spelling correction is performed to produce the output text sequence based at least on the determination that spelling correction against the ASR hypothesis should be triggered; as well as The ASR hypothesis is output as the output text sequence, based at least on the premise that spelling corrections for the ASR hypothesis are not triggered.

12. The computer storage device of claim 9, wherein the operation further comprises: Training a contextual spell checker, wherein the training includes generating text-to-speech (TTS) audio using preprocessed transcription with contextual phrases.

Images