Speech recognition using primary and secondary speech recognition engines

Abstract

The disclosed speech recognition techniques improve user-perceived latency while maintaining accuracy by receiving an audio stream in parallel by a primary (e.g., accurate) speech recognition engine (SRE) and a secondary (e.g., fast) SRE; generating primary results with the primary SRE; generating secondary results with the secondary SRE; appending the secondary results to a word list; and merging the primary results into the secondary results in the word list. Combining the outputs from the primary and secondary SREs into a single decoder as described herein improves user-perceived latency while maintaining or improving accuracy, among other advantages.

Description

Background Technology

[0001] Speech recognition typically uses a lookahead buffer for processing to improve accuracy. The longer the lookahead buffer, the later the speech recognition results are displayed to the user. The time difference between the time it takes to speak a word and the time it takes to recognize and display it to the user is the latency period. A long latency period can make the speech recognition engine (SRE) seem sluggish, leading to user dissatisfaction. However, shortening the lookahead buffer can negatively impact accuracy and also cause user dissatisfaction. Summary of the Invention

[0002] The disclosed examples are described in detail below with reference to the accompanying drawings. The following summary is provided to illustrate some of the examples disclosed herein. However, this is not to imply that all examples are limited to any particular configuration or sequence of operations.

[0003] The disclosed speech recognition technology improves perceived latency while maintaining accuracy by at least the following steps: An audio stream is received in parallel by a primary (e.g., precise) speech recognition engine (SRE) and a secondary (e.g., fast) SRE. A primary result is generated using the primary SRE. A secondary result is generated using the secondary SRE. The secondary result is appended to a word list. The primary result is merged into the secondary result in the word list. The merging includes, for example,: synchronizing the primary and secondary results; determining whether at least some words belong to a class model within the primary or secondary result; determining the words in the primary result that correspond to the corresponding words in the secondary result; and replacing the corresponding words in the secondary result with the words in the primary result if the corresponding words in the secondary result do not belong to a class model. Attached Figure Description

[0004] The disclosed examples are described in detail below with reference to the accompanying drawings:

[0005] Figure 1 The diagram illustrates a speech recognition setup that effectively reduces perceived latency while maintaining accuracy.

[0006] Figure 2 The diagram shows Figure 1 Further details of the layout;

[0007] Figure 3 The diagram shows Figure 1 Further details of some examples of the layout;

[0008] Figure 4 It also has an illustration. Figure 1 Further details of some examples of the layout;

[0009] Figure 5 The diagram shows that it can be used Figure 1 Example networks used within the layout;

[0010] Figure 6 It is illustrated with Figure 1 A flowchart of exemplary operations associated with the arrangement;

[0011] Figure 7 It is illustrated with Figure 6 A flowchart providing further details of the associated exemplary operations;

[0012] Figure 8 It is illustrated with Figure 1 Another flowchart illustrating the exemplary operations associated with the arrangement; and

[0013] Figure 9 This is a block diagram of an example computing environment suitable for implementing some of the various examples disclosed in this paper.

[0014] Throughout all the accompanying drawings, corresponding reference numerals indicate the corresponding components. Detailed Implementation

[0015] Various examples will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numerals are used in all drawings to denote the same or similar parts. References relating to specific examples and implementations throughout this disclosure are provided for illustrative purposes only, but are not intended to limit all examples unless otherwise indicated.

[0016] Various aspects of this disclosure advantageously improve the speed and accuracy of speech recognition by incorporating the results of a primary (e.g., high-accuracy) speech recognition engine (SRE) into the results of a secondary (e.g., low-latency) SRE in a word list, thereby reducing user-perceived latency in speech recognition while maintaining high accuracy. This high-accuracy, low-latency combination enhances performance compared to using only high-accuracy, long-latency SREs or only short-latency, low-accuracy SREs. This method eliminates the traditional trade-off between sacrificing the accuracy of rapid transcription results or suffering from delayed responses in order to achieve high accuracy. Various aspects of this disclosure also operate in an unconventional manner, at least based on determining that the corresponding words in the secondary results do not belong to a class model, by utilizing words from the primary results to replace the corresponding words in the secondary results. This special treatment of words found to belong to a class model, as described below, can further improve accuracy. Therefore, using multiple SREs as described herein reduces user-perceived latency. Furthermore, in the hybrid model example, processing speed on at least one device is increased, and the model size on the device is reduced, thereby using less storage.

[0017] Multiple SREs can operate in parallel, with audio signals input to them synchronously. The primary SRE uses a high-accuracy model, which may have high latency, while the secondary SRE uses a low-latency model, which may have lower accuracy. During speech recognition processing, the user sees incremental intermediate results as they speak and then sees the final result as the spoken phrase is completed. Intermediate results from the secondary SRE are generated with low latency to reduce perceived delay for the user, and the final result is generated from the primary SRE with high accuracy. The final result persists for the user and, in some examples, is used for voice commands, dictation, storage, transcription, further processing, or other purposes.

[0018] The merging process allows for seamless use of results from both the primary SRE and the secondary SRE, with the secondary SRE often leading the primary SRE in the reported results. The primary SRE results are concatenated to the word list, replacing earlier-issued words as needed to improve accuracy, while the word list can grow rapidly with low latency. For words belonging to a class model (e.g., words that can be checked against external references, such as contact list names and other common words, phrases, or names), the concatenation is paused. Furthermore, in some examples, the concatenation is paused across grammatical models (statistical models used for word sequences). In some examples of the operation, the secondary SRE issues a partial result. The partial result from the primary SRE is stored but not issued, and the merging algorithm fully replicates the stored primary partial result. The merging algorithm finds the word boundaries of the final word in this partial result and appends words from the secondary result to words whose starting word boundaries are greater than or equal to the previously found word boundaries. The resulting partial result is displayed to the user. The partial result is shown in real-time, while the final result appears when the speech stops. In some examples, merging is performed on the partial result rather than the final result.

[0019] The primary and secondary SREs can be tuned independently, providing flexibility in the trade-off between accuracy and latency. The secondary (e.g., often faster) SRE can be application-specific, for example, with processing that favors outputting certain terms (e.g., medical terms, financial terms, legal terms, gaming terms, and other technical terms). The secondary SRE uses a fast acoustic model (AM) and encodes sequences including future output states, generating probability distributions beyond those of a senone. A senone is a cluster of shared Hidden Markov Model (HMM) states, where each state shares the output probability distribution. For example, a single HMM models the relationship between input audio frames and triphones (phonemes in the context of other phonemes). An HMM has multiple states, a transition probability matrix, a start state, and an end state. Different states across a set of Markov models can share the output probability distribution. Sharing the output probability distribution increases the amount of training data assigned to it, thus improving the robustness of its parameter estimation. Such a cluster is called a senone. The output of a neural network (NN) (e.g., a deep NN (DNN) in a DNN-HMM hybrid decoder) includes the probability distribution of the supported senones (triplets of phonemes appearing in the sequence). For a given input audio frame, the neural network outputs the posterior probabilities of the senones in the senone set of the chosen acoustic model. The encoded sequence is then sent to a decoder, which decodes the sequence to output words.

[0020] The primary SRE uses a potentially slower AM and also encodes the sequence to generate a probability distribution on senone. In some examples, the encoded sequence from the fast AM (including future output states) is combined with the encoded sequence from the AM of the primary SRE. The jointly encoded sequence is then fed to the primary SRE decoder to output words with higher accuracy. In some examples, the primary SRE resides on a remote node on the network (e.g., in a cloud service provider network) to utilize higher power processing and larger storage, while the secondary SRE remains local.

[0021] Figure 1An arrangement 100 for speech recognition is illustrated, which advantageously improves user-perceived latency while maintaining accuracy. An audio stream 102 is captured from a user 106 via a microphone 104. The current time in the audio stream 102 is indicated by a timestamp 108. The arrangement 100 generates a word list 110, which may be a transcription of a live conversation (e.g., captions for a video conference or video stream, or captions for a voice command). The latest result (word or letter) in the word list 110 is displayed at a time indicated by a timestamp 112, which lags behind timestamp 108. This causes a user-perceived latency 113. A more accurate result is appended to the word list 110 at a time indicated by a timestamp 114, which lags behind timestamp 112 by another time delay 115. If not for the advantageous teachings herein, the user-perceived latency may be a potential latency 116, which is a combination of the user-perceived latency 113 plus the other time delays 115. Because the user-perceived latency 113 is shorter than the potential latency 116, this disclosure is able to improve the user-perceived latency while maintaining accuracy.

[0022] Audio stream 102 is input to Automatic Speech Recognition (ASR) Feature Extraction (FE) stage 118 and sent in parallel to primary SRE 120 and secondary SRE 130. Primary SRE 120 prioritizes high accuracy at the expense of potentially higher latency (slower performance), while secondary SRE 130 prioritizes low latency at the expense of potentially lower accuracy. Primary SRE 120 has a lookahead buffer 122, and secondary SRE 130 also has a lookahead buffer 132, although in some examples, lookahead buffer 122 is longer to provide higher accuracy, while lookahead buffer 132 is shorter to provide higher speed.

[0023] The main SRE 120 and auxiliary SRE 130 may include machine learning (ML) models, such as neural networks (NNs). In some examples, the main SRE 120 and / or auxiliary SRE 130 use an attention-based model that uses an encoder network to map the input acoustics to a higher-level representation and an attention-based decoder that conditioned the next output symbol (e.g., a word, phrase, or letter) on the previously predicted sequence. As illustrated, the main SRE 120 includes a main decoder 124 and a main encoder 126 as part of the main AM 128, and the auxiliary SRE 130 includes an auxiliary decoder 134 and an auxiliary encoder 136 as part of the auxiliary AM 138. Figure 3 and Figure 5 More details about the main SRE 120 and auxiliary SRE 130, as well as their respective components, are provided in the document.

[0024] The primary SRE 120 generates a primary result 140, which indicates word 142, and in some examples, multiple words 142, such as phrases. In some examples, the primary result 140 also includes a start synchronization flag 144, a stop synchronization flag 145, a class label 146, and a syntax label 148. Similarly, the secondary SRE 130 generates a secondary result 150, which indicates word 152, and in some examples, multiple words 152, such as phrases and / or individual letters. In some examples, the secondary result 150 also includes a start synchronization flag 154, a stop synchronization flag 155, a class label 156, and a syntax label 158. The start synchronization flag 144, start synchronization flag 154, stop synchronization flag 145, and stop synchronization flag 155 can be based on a timer 178, which also ensures that the primary SRE 120 and the secondary SRE 130 are synchronized (or, in some examples, separate timers are used). In some examples, the synchronization flags include timestamps.

[0025] Class tags 146 and 156 can be class start tags, class stop tags, or other types of tags indicating a class. In some examples, class tags are used to pause splicing so that identified words belonging to class model 170 are not spliced ​​(e.g., not changed or replaced). Class model 170 includes words that may need to be checked against references to words the user might use, such as names in contact list 172, application name 174, and other words 176. Other words can be times, dates, locations (place names), and generic names, such as the names of individuals or sports teams. Class model 170 can be grouped into sets of words with similar themes. Syntax tags 148 and 158 are used to prevent splicing across syntactic models, for example, splicing across different phrases. Figure 2 The description provides more details about pause splicing within a class model or across grammar models.

[0026] The main result 140 and the auxiliary result 150 are provided to merging logic 160, which combines the results to produce a word list 110. Merging logic 160 includes concatenation logic 162, which concatenates the main result 140 into the auxiliary result 150. In some examples, the concatenation occurs at the word level, rather than as part words. In some examples, the concatenation also occurs at the phrase level (multiple words), and concatenation across the grammar model is not desirable. About Figure 7 More details are provided regarding the merging logic 160 and the concatenation logic 162. For example... Figure 1As shown, the merging logic 160 has already appended the auxiliary result 150 to the word list 110 and is concatenating word 142 into the word list 110 to replace word 152. This occurs because words 142 and 152 correspond according to the start synchronization flag 144, start synchronization flag 154, stop synchronization flag 145, and stop synchronization flag 155. That is, words 142 and 152 appear simultaneously.

[0027] In some examples, the secondary SRE 130 sufficiently precedes the primary SRE 120, such that before word 142 of the future primary result 140 is appended to the secondary result 150 (e.g., at time marker 114), the secondary SRE 130 has already generated a later secondary result 150a (with at least one word 152a), and the merging logic 160 has already appended the later secondary result 150a to the word list 110 (e.g., at time marker 112). In some examples, the primary result 140 and the secondary result 150 represent different time periods within the audio stream 102, and thus different portions of the word list 110. In some examples, the primary result 140 and the secondary result 150 almost coincide in time. In some examples, the secondary result 150 precedes the primary result 140 by three or four words.

[0028] The tags described in this article can take any form, such as markup language tags (e.g., Extensible Markup Language tags).

[0029] Figure 2 Further details of word list 110 are illustrated. As indicated, secondary result 150 is reflected within word list 110, although in some examples, it is not indicated that all elements in secondary result 150 actually exist within word list 110. In some examples, some labels in the tags are preserved by merge logic 160 to identify whether they are concatenated in the various parts of main result 140, and word list 110 contains words and a limited set of metadata (e.g., timestamps). That is, in some examples, labels are removed before word list 110 is displayed.

[0030] As an example, audio stream 102 contains “Hey, Assistant. What's the weather?”. This has some notable features. “Hey Assistant” and “What's the weather?” are different phrases. These correspond to different grammatical models, so concatenation might occur within “Hey Assistant” or “What's the weather?”, but not across “Assistant. What’s.” Additionally, Assistant is the name of the application responding to voice commands, and therefore belongs to class model 170, specifically application name 174. For this example, both the secondary SRE 130 and the primary SRE 120 correctly identify the two grammatical models, and both correctly identify “Assistant” as belonging to the grammatical model. However, the secondary SRE 130 incorrectly identifies “What’s” as “Where’s”.

[0031] In auxiliary result 150, word W1 is "Hey", word W2 is "Assistant", word W3 is "Where's", word W4 is "the", and word W5 is "weather". In main result 140, word W1' is "Hey", word W2' is "Assistant", word W3' is "What's", word W4 is "the", and word W5 is "weather". In this example, word W3' means... Figure 1 Word 142, and word W3 represents the corresponding word 152. Word W3 (word 152) "Where's" will be replaced by word W3' (word 142) "What's".

[0032] Auxiliary result 150 has a grammatical start tag GS1 that begins the grammatical model “Hey, Assistant” and a grammatical stop tag GT1 that stops the grammatical model. Auxiliary result 150 also has a grammatical start tag GS2 that begins the grammatical model “Where's the weather?” and a grammatical stop tag GT2 that stops the grammatical model. The grammatical tags GT1 and GS2 will cause pauses during concatenation, limiting the concatenation to occur between grammatical tags GS1 and GT1 and between grammatical tags GS2 and GT2. The grammatical tags GS1, GT1, GS2, and GT2 are Figure 1Example of syntax tag 158. Main result 140 has a syntax start tag GS1' that begins the syntax model “Hey, Assistant” and a syntax stop tag GT1' that stops the syntax model. Main result 140 also has a syntax start tag GS2' that begins the syntax model “Where's the weather?” and a syntax stop tag GT2' that stops the syntax model. Syntax tags GT1' and GS2' will cause pauses in the concatenation, limiting the concatenation to occur between syntax tags GS1' and GT1' and between syntax tags GS2' and GT2'. Syntax tags GS1', GT1', GS2', and GT2' are Figure 1 Example of syntax tag 148. In some examples, the concatenation is paused by syntax tags in the main result 140 or the secondary result 150.

[0033] Auxiliary result 150 has a start synchronization flag SS1 and a stop synchronization flag ST1 for word W1, a start synchronization flag SS2 and a stop synchronization flag ST2 for word W2, a start synchronization flag SS3 and a stop synchronization flag ST3 for word W3, a start synchronization flag SS4 and a stop synchronization flag ST4 for word W4, and a start synchronization flag SS5 and a stop synchronization flag ST5 for word W5. Similarly, main result 140 has a start synchronization flag SS1' and a stop synchronization flag ST1' for word W1', a start synchronization flag SS2' and a stop synchronization flag ST2' for word W2', a start synchronization flag SS3' and a stop synchronization flag ST3' for word W3', a start synchronization flag SS4' and a stop synchronization flag ST4' for word W4', and a start synchronization flag SS5' and a stop synchronization flag ST5' for word W5'. The start and stop synchronization flags mark the beginning and end of words, enabling the merge logic 160 to determine which words in the main result 140 and the auxiliary result 150 are equivalent (e.g., appearing in the same time window). Start synchronization flags SS1 through SS5 are examples of start synchronization flag 154, stop synchronization flags ST1 through ST5 are examples of stop synchronization flag 155, start synchronization flags SS1' through SS5' are examples of start synchronization flag 144, and stop synchronization flags ST1' through ST5' are examples of stop synchronization flag 145.

[0034] In some examples, the synchronization flags vary slightly, thus allowing the merging logic 160 some timing precision tolerance. In some scenarios, the secondary SRE 130 and the primary SRE 120 can interpret the audio stream 102 differently, such that one recognizes two short words while the other recognizes a single, longer word. Adhering to the higher accuracy of the primary SRE 120, one-to-many word parsing can be performed using the earliest and latest synchronization flags, without concatenation restrictions within the class model, and duplicate words can also be detected and removed. In some scenarios, the (typically) more accurate primary SRE 120 recognizes additional words. For example, the secondary SRE 130 recognizes "John," while the primary SRE 120 recognizes "John Smith." If both recognized words are outside the class model (e.g., neither word belongs to the class model), then "Smith" will be concatenated into the word list 110 following "John."

[0035] Because "Assistant" belongs to class model 170, in auxiliary result 150, class start label CS1 precedes word W2, and class stop label CT1 follows word W2. In class models containing multiple words, class start label CS1 will precede the initial word, and class stop label CT1 will follow the final word. When merge logic 160 encounters class start label CS1, the concatenation is paused until after class stop label CT1. Similarly, main result 140 has class start label CS1' before word W2' and class stop label CT1' after word W2'. In some examples, class models are detected by main SRE 120 or auxiliary SRE 130, and concatenation will pause based on class start label CS1 or class start label CS1'. Class labels CS1 and CT1 are examples of class label 156, and class labels CS1' and CT1' are examples of class label 146.

[0036] Figure 3Further details of the main SRE 120 and auxiliary SRE 130, including a joint encoder 326b, are illustrated as used in some examples. In some examples, the main SRE 120 uses a larger main AM 128, while the auxiliary AM 138 in the auxiliary SRE 130 is smaller and simpler in structure. In some examples, to save memory space (or other computational burdens), the main SRE 120 and auxiliary SRE share some resources (e.g., shared resource 302). The shared resource 302 may include memory space and even NN components. As illustrated, the main SRE includes a main decoder 124, although the main encoder 126 is now shown as having two parts: a pre-encoder 326a and a joint encoder 326b. The auxiliary encoder 136 is fed to the joint encoder 326b. For a main AM 128 using encoding with six hidden layers, the pre-encoder 326a may have four layers. The pre-encoder 326a produces encoded sequences (senones) and can be structurally richer than the auxiliary encoder 136. In some examples, the main AM 128 uses a unidirectional long short-term memory (LSTM) network, a bidirectional LSTM network, or a different network architecture.

[0037] LSTM networks are a form of recurrent neural network (RNN) architecture with feedback connections, and are used for classification, processing, and prediction based on time series data because there may be lags of unknown duration between important events in a time series. Bidirectional LSTM networks use LSTM networks traveling in opposite directions, one taking input in the forward direction and the other in the backward direction. The main AM 128 and auxiliary AM 138 can have different architectures; for example, the main AM 128 uses a hybrid model, while the auxiliary AM 138 uses a recurrent neural network transformer (RNN-T) model, such as... Figure 5 As shown in the diagram. In some examples, the size of the secondary AM 138 is one-third to one-quarter the size of the primary AM 128.

[0038] In some examples, the encoded output of the co-encoder 136 is fed to a joint encoder 326 in one of the hidden layers. The output of the co-encoder 136 can lead the previous encoder 326a by several frames, thus enabling the co-encoder 136 to encode its current state and some future output states. The output of the co-encoder 136 can be concatenated to the output of the previous encoder 326a and fed to the joint encoder 326b. In some examples, the joint encoder 326b performs a softmax operation, which transforms the input set into values ​​between 0 and 1, making them interpretable as probabilities. In some scenarios, this joint encoding has the potential to produce more accurate speech recognition results.

[0039] In some examples, merge logic 160 and / or concatenation logic 162 use rule tree metadata to identify the subgrammar from which each word originates, in order to apply additional merge constraints and further improve the user experience. About Figure 7 More details are provided regarding the operation of merge logic 160 and / or splice logic 162.

[0040] Figure 4 The illustration shows version 400 of setup 100, where user 106 is using microphone 104 to capture audio stream 102 from local node 402 (e.g., a smartphone, tablet, personal computer (PC), or other computing device), and primary SRE 120 is executed on remote node 404 across network 930. Secondary SRE 130 remains on local node 402. In some examples, local node 402 and remote node 404 each include... Figure 9 The computing device 900. Local node 402 transmits audio stream 102 along with its timing information to remote node 404 (so that the synchronization flag can be accurate) for processing by master SRE 120. Remote node 404 returns a main result 140 for use by merging logic 160 to generate output for display element 410 (e.g., ...). Figure 9 The word list 110 is displayed as part of the presentation component 916 (such as a video screen). By viewing the display component 410, the user 106 only perceives a delay 113.

[0041] For example, version 400 can be used when the primary SRE 120 is too large or when performing computations on local node 402 is too computationally demanding. In some examples, the primary SRE 120 is four or five times the size of the secondary SRE 130 and requires significantly more computational power. If local node 402 is a small, portable, battery-powered device, the performance of arrangement 100 may be compromised if the primary SRE 120 performs locally. However, in some examples of arrangement 110, both the primary SRE 120 and the secondary SRE reside on a single computing device (e.g., computing device 900 or local node 402), which could be a smartphone, tablet, or PC, such as a desktop or laptop PC.

[0042] Figure 5An exemplary recurrent neural network converter (RNN-T) 500 is illustrated that can be used within arrangement 100 (e.g., within the auxiliary SRE 130 as noted above). That is, in some examples, the auxiliary SRE includes an RNN-T 500. The RNN-T 500 processes input samples and streams output symbols, which are used for speech dictation. In some examples, the output symbols are words or alphabetic characters. The RNN-T 500 recognizes the output one word at a time as it receives speech, with whitespace between words.

[0043] During operation, an audio segment 502, as part of an audio stream 102, is input to an encoder 504, which can be used as an auxiliary encoder 136. A joint network 506 receives the outputs of the encoder 504 and a prediction network 508, which makes predictions based on a previous output 516. That is, with a time reference where the previous output 516 is "current" and the output of the joint network 506 is a "future" output, the prediction network 508 predicts the future output based on the current output. The output of the joint network 506 is subject to a softmax operation 510, which transforms the input set into values ​​between 0 and 1, making them interpretable as probabilities. The RNN-T 500 outputs 512, which is copied to the previous output 516 via a feedback loop along with the next audio segment 502 for use in the next iteration.

[0044] Figure 6 This is a flowchart 600 illustrating exemplary operations included in performing speech recognition, which advantageously improves user-perceived latency while maintaining accuracy. In some examples, the operations described in flowchart 600 are performed by… Figure 9 The computing device 900 is used for execution. It should utilize... Figures 1 to 5 Check Figure 6 .

[0045] Flowchart 600 begins with operation 602, which includes receiving audio stream 102 in parallel by a primary SRE 120 and a secondary SRE 130. In some examples, operation 602 includes receiving audio stream 102 by the secondary SRE 130 on a local node 402 (computing device) and transmitting audio stream 102 to a remote node 404 for processing by the primary SRE 120. In some examples, timing information of audio stream 102 is also transmitted to ensure accurate synchronization flags. Operation 604 includes generating secondary result 150 using the secondary SRE 130. In some examples, the secondary SRE 130 includes a secondary encoder 136 and a secondary decoder 134. In some examples, the secondary SRE 130 includes an ML model. In some examples, the secondary SRE 130 includes a neural network. In some examples, the secondary encoder 136 outputs a probabilistic senone. The secondary decoder 134 outputs a word (e.g., word 152). In some examples, the auxiliary result 150 includes the identified word 152, a start synchronization flag 154, a stop synchronization flag 155, and a class label 156. In some examples, the synchronization flags (e.g., start synchronization flag 154 and stop synchronization flag 155) include a timestamp. In some examples, the class label 156 includes a class start label or a class stop label.

[0046] As part of operation 604 (in some examples), operation 606 includes: within auxiliary result 150, determining whether at least some words belong to class model 170. In some examples, if so, class labels are applied before and after the words belonging to class model 170 (see [link to example 150]). Figure 2 In some examples, class model 170 is selected from a list that includes the following items: contact name, date, time, application name, file name, location, and commonly identified name. In some examples, the commonly identified name includes the name of a sports team or the name of a person.

[0047] Operation 608 includes generating a main result 140 using the main SRE 120. In some examples, the main SRE 120 has higher accuracy than the auxiliary SRE 130, at the cost of potentially higher latency. In some examples, the auxiliary SRE 130 has lower latency than the main SRE 120, at the cost of potentially lower accuracy. In some examples, the main SRE 120 includes a main encoder 126 and a main decoder 124. In some examples, the main SRE 120 includes an ML model. In some examples, the main SRE 120 includes a neural network. In some examples, the main encoder 126 outputs a senone with a probability distribution. The main decoder 124 outputs a word (e.g., word 142). In some examples, the main result 140 includes the identified word 142, a start synchronization flag 144, a stop synchronization flag 145, and a class label 146. In some examples, the synchronization flags (e.g., start synchronization flag 144 and stop synchronization flag 145) include timestamps. In some examples, class label 146 includes a class start label or a class stop label. In some examples, operation 608 also includes operation 610, providing the encoded sequence from the secondary SRE 130 to an intermediate level of the primary SRE 120 (e.g., Figure 3 (as shown in the image).

[0048] Operation 612 includes determining, within the main result 140, whether at least some words belong to class model 170, also as part of operation 608 (in some examples). Operation 614 includes receiving the main result 140 from the remote node 404 across network 930 when using version 400 of layout 100 (e.g., the primary SRE 120 resides on a remote node 404, while the secondary SRE 130 resides on a local node 402).

[0049] Some examples of layout 100 append word list 110 to secondary result 150, while some examples of layout 100 append word list 110 where either secondary result 150 or primary result 140 precedes (e.g., in time). Although secondary SRE 130 may generally be faster than primary SRE 120, in some scenarios, primary SRE 120 may actually precede secondary SRE 130. Therefore, operation 616 includes determining whether secondary SRE 130 precedes primary SRE 120 or whether primary SRE 120 precedes secondary SRE 130. This can be achieved using synchronization flags 144, 145, 154, and 155. Operation 618 includes appending the result to word list 110. In some examples, the default is to append secondary result 150 to word list 110. In some examples, operation 618 appends secondary result 150 to word list 110 when secondary result 150 precedes primary result 140. In such an example, appending the secondary result 150 to the word list 110 includes appending the secondary result 150 to the word list 110 based on at least the secondary SRE 130 leading the primary SRE 120. However, in some examples, operation 618 includes appending the primary result 140 to the word list 110 based on at least the primary SRE 120 leading the secondary SRE 130.

[0050] Operation 620 includes merging the main result 140 into the auxiliary result 150 in the word list 110. Operation 620 includes operations 622 to 628 and is an ongoing loop operation within loop operations 602 to 634. In some examples, the merging operation 620 does not occur across different grammatical models. Operation 622 includes synchronizing the main result 140 with the auxiliary result 150. In some examples, synchronizing the main result 140 with the auxiliary result 150 includes comparing the synchronization flags of the main result 140 with the synchronization flags of the auxiliary result 150 (e.g., comparing the start synchronization flag 144 with the start synchronization flag 154 and / or comparing the stop synchronization flag 145 with the stop synchronization flag 155).

[0051] Operation 624 includes: determining, at least based on synchronization, the word 142 in the main result 140 corresponding to the corresponding word 152 in the auxiliary result 150. Determination operation 626 includes: determining whether the word 142 in the main result 140 is different from the corresponding word 152 in the auxiliary result 150. If determination operation 626 determines that there is no difference, some examples abandon the concatenation and proceed to the next word or grammar model. Determination operation 628 includes: determining whether the class model or grammar model pauses the concatenation. For example, determination operation 628 includes: within the main result 140 or auxiliary result 150, determining whether word 142 or corresponding word 152 belongs to class model 170. Additionally, determination operation 628 may include: determining whether the concatenation will occur within the grammar model (allowed) or across grammar categories (prohibited in some examples). If the concatenation is paused, operation 620 returns to 622 or 624.

[0052] If concatenation is permitted (e.g., without pauses), operation 630 includes: replacing the corresponding word 152 in the secondary result 150 with the word 142 in the primary result 140, based on at least determining that the corresponding word 152 in the secondary result 150 does not belong to the category model 170. In some examples, replacing the corresponding word 152 in the secondary result 150 with the word 142 in the primary result 140 includes: replacing the corresponding word 152 in the secondary result 150 with the word 142 in the primary result 140, based on at least determining that the word 142 in the primary result 140 is different from the corresponding word 152 in the secondary result 150 and determining that the corresponding word 152 in the secondary result 150 does not belong to the category model 170.

[0053] Operation 632 includes: displaying word list 110, for example, displaying word list 110 to user 106 on display element 410, for example, as captions for a video stream or video conference. In some examples, word list 110 includes at least a portion of a live transcript of a live session. Operation 634 includes: using words from the identified word list 110 as audio commands. Operations 602 to 634 form an ongoing loop, with operation 220 looping internally.

[0054] Figure 7 The diagram shows that it can be used with Figure 6 The flowchart 600 is used in conjunction with the exemplary operation flowchart 700. In some examples, the operation described in flowchart 700 is performed by... Figure 9The computational device 900 performs the operation. Flowchart 700 begins with operation 702, in which the auxiliary SRE 130 issues an auxiliary result 150, which may be a partial result. In operation 704, the main SRE 120 generates a main result 140, which may also be a partial result, and stores the main result 140. In operation 706, merge logic 160 copies the stored main result 140 and finds the word boundary of the final word in the main result 140 (e.g., using the final stop synchronization flag 145). In operation 708, merge logic 160 appends words from the auxiliary result 150 whose start word boundary (e.g., the earliest start synchronization flag 145) is greater than or equal to (e.g., using the last stop synchronization flag 145) the word boundary found for the main result 140. This corresponds to operation 618 of flowchart 600. In operation 710 (corresponding to operation 632 of flowchart 600), the result is displayed to user 106 as a word list 110. In some examples, partial results are used instead of waiting for the final result to achieve merging, in order to reduce latency. Merging uses synchronization flags (e.g., word timestamps) and can use rule tree metadata to identify the subgrammar from which each word comes, in order to apply additional merge constraints and further improve the user experience.

[0055] Figure 8 This is a flowchart 800 illustrating exemplary operations included in performing speech recognition, which advantageously improves user-perceived latency while maintaining accuracy. In some examples, the operations described in flowchart 800 are performed by… Figure 9 The computational device 900 performs the operation. Flowchart 800 begins with operation 802, which includes: receiving an audio stream in parallel by a main SRE and a secondary SRE. Operation 804 includes: generating a main result using the main SRE. Operation 806 includes: generating a secondary result using the secondary SRE. Operation 804 includes: appending the secondary result to a word list. Operation 810 includes: merging the main result into the secondary result in the word list, which includes operations 812 to 818.

[0056] Operation 812 includes synchronizing the main result and the auxiliary result. Operation 814 includes determining, within the main result or the auxiliary result, whether at least some words belong to the class model. Operation 816 includes determining, at least based on synchronization, the words in the main result that correspond to the corresponding words in the auxiliary result. Operation 818 includes, at least based on determining that the corresponding words in the auxiliary result do not belong to the class model, replacing the corresponding words in the auxiliary result with the words in the main result.

[0057] Additional examples

[0058] An example speech recognition method includes: receiving an audio stream in parallel by a main SRE and a secondary SRE; generating a main result using the main SRE; generating a secondary result using the secondary SRE; appending the secondary result to a word list; and merging the main result into the secondary result in the word list, wherein merging includes: synchronizing the main result with the secondary result; determining, within the main result or the secondary result, whether at least some words belong to a class model; determining, at least based on synchronization, words in the main result that correspond to corresponding words in the secondary result; and at least based on determining that corresponding words in the secondary result do not belong to a class model, replacing corresponding words in the secondary result with words in the main result.

[0059] An example speech recognition system includes a processor and a computer-readable medium storing instructions that, when executed by the processor, are operable to: receive an audio stream in parallel by a primary SRE and a secondary SRE; generate a primary result using the primary SRE; generate a secondary result using the secondary SRE; append the secondary result to a word list; and merge the primary result into the secondary result in the word list, wherein merging includes: synchronizing the primary result with the secondary result; determining, within the primary or secondary result, whether at least some words belong to a class model; determining, at least based on synchronization, words in the primary result that correspond to corresponding words in the secondary result; and replacing corresponding words in the secondary result with words in the primary result, at least based on determining that corresponding words in the secondary result do not belong to a class model.

[0060] One or more example computer storage devices (e.g., computing devices) storing computer-executable instructions that, when executed by a computer, cause the computer to perform operations including: receiving an audio stream in parallel by a primary speech SRE and a secondary SRE; generating a primary result using the primary SRE; generating a secondary result using the secondary SRE; appending the secondary result to a word list; and merging the primary result into the secondary result in the word list, wherein merging includes: synchronizing the primary result with the secondary result; determining, within the primary or secondary result, whether at least some words belong to a class model; determining, at least based on synchronization, words in the primary result that correspond to corresponding words in the secondary result; and replacing corresponding words in the secondary result with words in the primary result, at least based on determining that corresponding words in the secondary result do not belong to a class model.

[0061] An example computing device stores computer-executable instructions that, when executed by a computer, cause the computer to perform operations including: receiving an audio stream by a secondary speech recognition engine (SRE) on the computing device; transmitting the audio stream to a remote node for processing by a primary SRE; receiving a primary result from the remote node; generating a secondary result using the secondary SRE; appending the secondary result to a word list; and merging the primary result into the secondary result in the word list, wherein merging includes: synchronizing the primary result with the secondary result; determining, within the primary or secondary result, whether at least some words belong to a class model; determining, at least based on synchronization, words in the primary result corresponding to corresponding words in the secondary result; and at least based on determining that corresponding words in the secondary result do not belong to a class model, replacing corresponding words in the secondary result with words in the primary result.

[0062] Alternatively, or in addition to the other examples described herein, examples may include any combination of the following:

[0063] - The accuracy of primary SRE is higher than that of secondary SRE;

[0064] - The latency of the secondary SRE is lower than that of the primary SRE;

[0065] - Displays a list of words;

[0066] - Use words from the recognized word list as voice commands;

[0067] - The word list includes at least a portion of the live transcripts of the live conversation;

[0068] - Receive master results from remote nodes across the network;

[0069] - The primary SRE resides on a remote node, while the secondary SRE resides on a local node;

[0070] Both the primary SRE and the secondary SRE reside on a single computing device;

[0071] - Provide the encoded sequence from the secondary SRE to the intermediate stage of the primary SRE;

[0072] - Synchronizing the main result with the auxiliary result includes: comparing the synchronization flag of the main result with the synchronization flag of the auxiliary result;

[0073] - Determine whether the words in the main result are different from the corresponding words in the secondary result;

[0074] - Replacing the corresponding words in the auxiliary result with words from the main result includes: based on at least determining that the words in the main result are different from the corresponding words in the auxiliary result, and determining that the corresponding words in the auxiliary result do not belong to the class model, replacing the corresponding words in the auxiliary result with words from the main result;

[0075] - Select a class model from the list, which includes contact name, date, time, application name, file name, location, and commonly identifiable name;

[0076] - The commonly identified names include the names of sports teams or individuals;

[0077] - The main result includes the identified word, start synchronization flag, stop synchronization flag, and class label;

[0078] - The auxiliary results include the identified words, the start synchronization flag, the stop synchronization flag, and the class label;

[0079] - Class tags include class start tags or class stop tags;

[0080] - Merging will not occur across different syntax models;

[0081] -The main SRE includes the main encoder and the auxiliary decoder;

[0082] - The main encoder outputs a probabilistic senone;

[0083] - Output words of the main decoder;

[0084] -The main SRE includes the ML model;

[0085] -The main SRE includes the NN;

[0086] - The auxiliary SRE includes an auxiliary encoder and an auxiliary decoder;

[0087] - The secondary encoder outputs a probabilistic senone;

[0088] - Output words of the auxiliary decoder;

[0089] - Auxiliary SREs include ML models;

[0090] - Auxiliary SRE includes NN;

[0091] - Secondary SREs include RNN-T;

[0092] - Synchronization flags include timestamps;

[0093] - Determine whether the secondary SRE is ahead of the primary SRE, or whether the primary SRE is ahead of the secondary SRE;

[0094] - Appending secondary results to the word list includes: at least based on the secondary SRE leading the primary SRE, appending secondary results to the word list; and

[0095] - Based on at least the main SRE preceding the auxiliary SRE, append the main result to the word list.

[0096] Although aspects of the invention have been described with reference to various examples having their associated operations, those skilled in the art will appreciate that combinations of operations from any number of different examples are also within the scope of the invention.

[0097] Example operating environment

[0098] Figure 9 This is a block diagram of an example computing device 900 used to implement the aspects disclosed herein, and is generally designated as computing device 900. Computing device 900 is merely an example of a suitable computing environment and is not intended to impose any limitation on the scope or functionality of the examples disclosed herein. Nor should computing device 900 be construed as having any dependency or requirement on any component / module or combination thereof illustrated. The examples disclosed herein can be described in the general context of computer code or machine-usable instructions (including computer-executable instructions such as program components) that are executed by a computer or other machine such as a personal data assistant or other handheld device. Generally, a program component, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. The disclosed examples can be practiced in a variety of system configurations, including personal computers, laptops, smartphones, mobile tablets, handheld devices, consumer electronics, dedicated computing devices, etc. The disclosed examples can also be practiced in distributed computing environments when tasks are performed by remote processing devices linked via a communication network.

[0099] Computing device 900 includes a bus 910 that directly or indirectly couples to the following devices: computer storage memory 912, one or more processors 914, one or more presentation components 916, I / O ports 918, I / O components 920, power supply 922, and network components 924. Although computing device 900 is depicted as appearing to be a single device, multiple computing devices 900 can work together and share the depicted device resources. For example, memory 912 is distributed across multiple devices, and one or more processors 914 are housed in different devices.

[0100] Bus 910 represents a bus that can be one or more buses (such as an address bus, a data bus, or a combination thereof). Although lines are used to illustrate for clarity... Figure 9 The various boxes are used to represent the components, but alternative representations can be used to depict each component. For example, a presentation component such as a display device is an I / O component in some examples, and some examples of processors have their own memory. No distinction is made between categories such as "workstation," "server," "laptop," "handheld device," etc., because all of these are considered to be within... Figure 9Within the scope of this document, and referred to herein as "computing device," memory 912 may take the form of computer storage media and is operable to provide storage for computer-readable instructions, data structures, program modules, and other data for computing device 900. In some examples, memory 912 stores one or more of the following: operating system, general-purpose application platform, or other program modules and program data. Therefore, memory 912 is capable of storing and accessing data 912a and instructions 912b, which can be executed by processor 914 and configured to perform the various operations disclosed herein.

[0101] In some examples, memory 912 includes volatile and / or non-volatile memory, removable or non-removable memory, a data disk in a virtual environment, or a combination thereof, as a computer storage medium. Memory 912 may include any number of memories associated with or accessible by computing device 900. Memory 912 may be internal to computing device 900 (e.g., Figure 9 The memory 912 may be located outside the computing device 900 (not shown), or both (not shown). Examples of memory 912 include, but are not limited to, random access memory (RAM); read-only memory (ROM); electrically erasable programmable read-only memory (EEPROM); flash memory or other memory technologies; CD-ROM, digital versatile disc (DVD) or other optical or holographic media; magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices; memory connected to an analog computing device; or any other medium used to encode desired information and accessed by the computing device 900. Additionally or alternatively, the memory 912 may be distributed across multiple computing devices 900, for example, in a virtualized environment where instruction processing is performed on multiple devices 900. For the purposes of this disclosure, “computer storage medium,” “computer storage memory,” “memory,” and “memory device” are synonymous terms for computer storage memory 912, and none of these terms includes a carrier or propagation signaling.

[0102] One or more processors 914 may include any number of processing units that read data from various entities such as memory 912 or I / O components 920. Specifically, processor 914 is programmed to execute computer-executable instructions for implementing aspects of this disclosure. These instructions may be executed by a processor, by multiple processors within computing device 900, or by a processor external to client computing device 900. In some examples, processor 914 is programmed to execute instructions such as those illustrated in the flowcharts discussed below and depicted in the accompanying drawings. Furthermore, in some examples, one or more processors 914 represent implementations of analog techniques for performing the operations described herein. For example, these operations are performed by analog client computing device 900 and / or digital client computing device 900. A presentation component 916 presents data indications to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, etc. Those skilled in the art will understand and appreciate that computer data can be presented in several ways, such as visually in a graphical user interface (GUI), audibly through a speaker, wirelessly between computing devices 900, across a wired connection, or otherwise. I / O port 918 allows computing device 900 to be logically coupled to other devices including I / O components 920, some of which may be built-in. Example I / O components 920 include, for example, but not limited to, microphones, joysticks, game controllers, satellite antennas, scanners, printers, wireless devices, etc.

[0103] Computing device 900 can operate in a networked environment via network component 924 using a logical connection to one or more remote computers. In some examples, network component 924 includes a network interface card and / or computer-executable instructions (e.g., drivers) for operating a network interface card. Communication between computing device 900 and other devices can occur via any wired or wireless connection using any protocol or mechanism. In some examples, network component 924 is operable to use short-range communication technologies (e.g., Near Field Communication (NFC), Bluetooth). TM Wireless devices (such as brand communication devices) or combinations thereof transmit data via public, private, or hybrid (public and private) protocols. Network component 924 communicates with cloud resource 928 across network 930 via wireless communication link 926 and / or wired communication link 926a. Various examples of communication links 926 and 926a include wireless connections, wired connections, and / or private links, and in some examples, at least a portion is routed over the Internet.

[0104] Although described in conjunction with example computing device 900, the examples of this disclosure can be implemented using many other general-purpose or special-purpose computing system environments, configurations, or devices. Examples of well-known computing systems, environments, and / or configurations suitable for use with aspects of this disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or communication devices in wearable or accessory form factors (e.g., watches, glasses, headsets, or receivers), network PCs, minicomputers, mainframes, distributed computing environments including any of the systems or devices described above, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, holographic devices, etc. Such systems or devices can accept input from users in any way, including from input devices such as keyboards or pointing devices, via gesture input, proximity input (such as by hovering), and / or voice input.

[0105] Examples of this disclosure can be described in the general context of computer-executable instructions, such as program modules, that are executed by one or more computers or other devices, including software, firmware, hardware, or combinations thereof. Computer-executable instructions can be organized into one or more computer-executable components or modules. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform a particular task or implement a particular abstract data type. Aspects of this disclosure can be implemented using any number and organization of such components or modules. For example, aspects of this disclosure are not limited to the specific computer-executable instructions or specific components or modules illustrated in the accompanying drawings and described herein. Other examples of this disclosure may include different computer-executable instructions or components that have more or fewer functions than those illustrated and described herein. In examples including general-purpose computers, aspects of this disclosure, when configured to execute the instructions described herein, transform a general-purpose computer into a special-purpose computing device.

[0106] By way of example and not limitation, computer-readable media include computer storage media and communication media. Computer storage media includes volatile and non-volatile memory, removable memory, and non-removable memory implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media are tangible and mutually exclusive with communication media. Computer storage media are implemented in hardware and do not include carrier waves and propagating signals. The computer storage media used in this disclosure are not signals themselves. Exemplary computer storage media include hard disks, flash drives, solid-state memory, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible to computing devices. In contrast, communication media typically embody computer-readable instructions, data structures, program modules, and any information delivery medium in the form of modulated data signals such as carrier waves or other transmission mechanisms.

[0107] The order in which operations are performed in the examples of this disclosure shown and described herein is not essential and may be performed in different orders in various examples. For example, it is within the scope of various aspects of the invention that a particular operation is contemplated to be performed before, simultaneously with, or after another operation. When introducing elements of aspects of this disclosure or its examples, the articles “a,” “an,” “the,” and “described” are intended to mean one or more of the elements present. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may be present in addition to the listed elements. The term “exemplary” is intended to mean “an example of…”. The phrase “one or more of the following: A, B, and C” means “at least one of A and / or at least one of B and / or at least one of C.”

[0108] Having described in detail all aspects of this disclosure, it will be apparent that modifications and variations are possible without departing from the scope of the aspects of this disclosure as defined in the appended claims. Since various changes can be made to the above-described structures, products, and methods without departing from the scope of the aspects of this disclosure, all content contained in the foregoing description and shown in the accompanying drawings should be interpreted as illustrative rather than restrictive.

Claims

1. A speech recognition method, the method comprising: The audio stream is received in parallel by the main speech recognition engine (SRE) and the auxiliary SRE. Generate a cascaded output, which includes the encoded output of the secondary SRE and the early-stage encoded output of the primary SRE; The cascaded output is processed by the main SRE; The encoded output of the auxiliary SRE is processed by the auxiliary SRE; The main result is generated using the processed output of the main SRE; Auxiliary results are generated using the processed output of the auxiliary SRE; Append the auxiliary results to the word list; as well as The main result is merged into the secondary result in the word list, wherein the merging includes: Synchronize the main result with the auxiliary result; Within the main result or the auxiliary result, at least some words are identified as belonging to the class model; Based at least on the synchronization, determine the word in the main result that corresponds to the corresponding word in the auxiliary result; and Based at least on the determination that the corresponding word in the auxiliary result does not belong to the class model, the corresponding word in the auxiliary result is replaced by the word in the main result.

2. The method according to claim 1, further comprising: Display the list of words.

3. The method according to claim 1, further comprising: Based at least on the determination that the corresponding word in the auxiliary result and the word in the main result belong to the same grammatical model, the corresponding word in the auxiliary result is replaced by the word in the main result.

4. The method of claim 1, wherein generating the cascaded output comprises: The encoding output of the secondary SRE is determined to be earlier in time than the encoding output of the previous stage, and the concatenation is performed based on this determination.

5. The method of claim 1, wherein synchronizing the main result with the auxiliary result comprises: The synchronization flag of the main result is compared with the synchronization flag of the auxiliary result.

6. The method according to claim 1, further comprising: Determining whether the word in the main result is different from the corresponding word in the auxiliary result, wherein replacing the corresponding word in the auxiliary result with the word in the main result includes: Based at least on the determination that the word in the main result is different from the corresponding word in the auxiliary result and that the corresponding word in the auxiliary result does not belong to the class model, the word in the main result is used to replace the corresponding word in the auxiliary result.

7. The method of claim 1, wherein the class model is selected from a list comprising: Contact name, date, time, application name, file name, location, and commonly identified name.

8. A system for speech recognition, the system comprising: processor; as well as A computer-readable medium storing instructions that, when executed by the processor, are operable to: The audio stream is received in parallel by the main speech recognition engine (SRE) and the auxiliary SRE. Generate a cascaded output, which includes the encoded output of the secondary SRE and the early-stage encoded output of the primary SRE; The cascaded output is processed by the main SRE; The encoded output of the auxiliary SRE is processed by the auxiliary SRE; The main result is generated using the processed output of the main SRE; Auxiliary results are generated using the processed output of the auxiliary SRE; Append the auxiliary results to the word list; as well as The main result is merged into the secondary result in the word list, wherein the merging includes: Synchronize the main result with the auxiliary result; Within the main result or the auxiliary result, at least some words are identified as belonging to the class model; Based at least on the synchronization, determine the word in the main result that corresponds to the corresponding word in the auxiliary result; and Based at least on the determination that the corresponding word in the auxiliary result does not belong to the class model, the corresponding word in the auxiliary result is replaced by the word in the main result.

9. The system of claim 8, wherein the auxiliary SRE comprises a recurrent neural network converter (RNN-T).

10. The system of claim 8, wherein the instructions are further operable to: Based at least on the determination that the corresponding word in the auxiliary result and the word in the main result belong to the same grammatical model, the corresponding word in the auxiliary result is replaced by the word in the main result.

11. The system of claim 8, wherein generating the cascaded output comprises: The encoding output of the secondary SRE is determined to be earlier in time than the encoding output of the previous stage, and the concatenation is performed based on this determination.

12. The system of claim 8, wherein synchronizing the main result with the auxiliary result comprises: The synchronization flag of the main result is compared with the synchronization flag of the auxiliary result.

13. The system of claim 8, wherein the instructions are further operable to: Determining whether the word in the main result is different from the corresponding word in the auxiliary result, wherein replacing the corresponding word in the auxiliary result with the word in the main result includes: Based at least on the determination that the word in the main result is different from the corresponding word in the auxiliary result and that the corresponding word in the auxiliary result does not belong to the class model, the word in the main result is used to replace the corresponding word in the auxiliary result.

14. The system of claim 8, wherein the class model is selected from a list including: Contact name, date, time, application name, file name, location, and commonly identified name.

15. A computing device having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a computer, causing the computer to perform operations, the operations including: The audio stream is received by the auxiliary speech recognition engine (SRE) on the computing device; The audio stream is transmitted to a remote node for processing by the main SRE; The encoded output of the secondary SRE is transmitted to the remote node, wherein the remote node generates a concatenated output including the encoded output of the secondary SRE and the early stage encoded output of the primary SRE, and further processes the concatenated output; Based on the further processing of the cascaded output, the main result is received from the remote node; The encoded output of the auxiliary SRE is further processed by the auxiliary SRE; The output of the auxiliary SRE is used for further processing to generate auxiliary results; Append the auxiliary results to the word list; as well as The main result is merged into the secondary result in the word list, wherein the merging includes: Synchronize the main result with the auxiliary result; Within the main result or the auxiliary result, at least some words are identified as belonging to the class model; Based at least on the synchronization, determine the word in the main result that corresponds to the corresponding word in the auxiliary result; and Based at least on the determination that the corresponding word in the auxiliary result does not belong to the class model, the corresponding word in the auxiliary result is replaced by the word in the main result.

16. The computing device of claim 15, wherein the operation further comprises: Display the list of words.

17. The computing device of claim 15, wherein generating the cascaded output comprises: The encoding output of the secondary SRE is determined to be earlier in time than the encoding output of the previous stage, and the concatenation is performed based on this determination.

18. The computing device of claim 15, wherein synchronizing the main result with the auxiliary result comprises: The synchronization flag of the main result is compared with the synchronization flag of the auxiliary result.

19. The computing device of claim 15, wherein the operation further comprises: Determining whether the word in the main result is different from the corresponding word in the auxiliary result, wherein replacing the corresponding word in the auxiliary result with the word in the main result includes: Based at least on the determination that the word in the main result is different from the corresponding word in the auxiliary result and that the corresponding word in the auxiliary result does not belong to the class model, the word in the main result is used to replace the corresponding word in the auxiliary result.

20. The computing device of claim 15, wherein the class model is selected from a list comprising: Contact name, date, time, application name, file name, location, and commonly identified name.

Images