System and method for streaming end-to-end speech recognition with asynchronous decoder

Abstract

The speech recognition system continuously processes each encoder state of the encoded acoustic features using a frame synchronization decoder (FSD) module and a label synchronization decoder (LSD) module. When an encoder state that carries information about a new transcription output is identified, the system uses the FSD module to expand the current FSD prefix list, uses the LSD module to evaluate the FSD prefix, and prunes the FSD prefix based on the combined FSD score and LSD score. The FSD module and the LSD module are synchronized by having the LSD module process the portion of the encoder state that includes the new transcription output identified by the FSD module and generate an LSD score for the FSD prefix determined by the FSD module.

Description

Technical Field

[0001] The present invention generally relates to a speech recognition system, and more particularly, to a method and system for streaming end-to-end speech recognition with joint frame synchronous decoding (FSD) and label synchronous decoding (LSD) for automatic speech recognition applications with low latency requirements. Background Art

[0002] Automatic speech recognition (ASR) systems are widely deployed for various interface applications such as speech-based search. In recent years, end-to-end and sequence-to-sequence neural network models have gained increasing attention and popularity in the ASR community. The output of an end-to-end ASR system is typically a grapheme sequence, which can be a single letter or a larger unit such as a word fragment, a complete word, or a sentence fragment. The appeal of end-to-end ASR lies in that compared to traditional ASR systems, end-to-end ASR achieves a simplified system structure because it mainly consists of neural network components and avoids the language expert knowledge required to build an ASR system. Such an end-to-end ASR system can directly learn all components of the speech recognizer, including the pronunciation model, acoustic model, and language model, which avoids the need for language-specific linguistic information and tokenization.

[0003] Sequence-to-sequence models for end-to-end ASR systems are mainly based on two types of neural network output paradigms, namely, neural networks with frame-by-frame decisions for frame-synchronized decoding (FSD) and neural networks with label-by-label decisions using label-synchronized decoding (LSD). End-to-end ASR systems with FSD are more suitable for streaming / online ASR applications, where ASR output can be generated immediately after each spoken word with low latency. However, ASR systems based on LSD (which often show excellent results in speech recognition) are less suitable for online / streaming ASR applications because they usually require the entire speech utterance as input due to the loss of alignment information (i.e., lack of information about whether the input sequence contains enough information to generate the next output token) before decoding.

[0004] Therefore, FSD and LSD are asynchronous decoders. Potentially, in applications where streaming end-to-end speech recognition is not required, the combination of FSD neural network and LSD neural network can achieve a lower word error rate than each single neural network model. However, these ASR systems require synchronization between the FSD neural network and the LSD neural network in order to enable the ASR system to achieve "real-time" speech recognition.

[0005] Therefore, there is a need to effectively synchronize the FSD neural network and the LSD neural network in order to achieve combined FSD-LSD based ASR for “real-time” / online / streaming applications. Summary of the Invention

[0006] Automatic speech recognition (ASR) systems face a sequence-to-sequence modeling problem in which an input sequence of audio samples or a sequence of acoustic features extracted from audio frames is mapped to an output sequence of characters. Some methods for such sequence-to-sequence mapping are limited to offline ASR applications, where the entire speech utterance is available for the recognition process. Such methods cannot be applied in online / streaming ASR systems with low latency constraints. It is an object of some embodiments to implement an ASR system for "real-time" applications by synchronizing the frame synchronized decoder (FSD) module and the label synchronized decoder (LSD) module. A streaming application can be any application that requires "real-time" transcription of a speech signal, for example, transcribing a speech utterance in an ongoing call, an ongoing lecture, an ongoing conversation, or detecting and recognizing voice commands. In addition, it is an object of some embodiments to improve the performance of the LSD module by reducing the output delay introduced by the LSD module and improving the accuracy of the utterance transcriptions generated by the FSD module as well as the LSD module.

[0007] Some embodiments are based on the recognition that LSD-based ASR systems need to observe a complete input sequence (which is typically an entire speech utterance segmented by speech pauses) in order to assign a weight to each element of the input sequence in order to identify each output label of the output sequence. For example, an output label may include a single alphabetic character or a sequence of characters, such as a word or a sentence fragment. Due to the lack of a priori knowledge about which parts of the input sequence are relevant to identifying the next output label and the need to assign a weight to each element of the input sequence, LSD-based modules typically need to process large input sequences. Such processing allows the advantages of focusing on different parts of the utterance to be exploited, but also increases output latency and is therefore impractical for speech recognition in a streaming / online manner.

[0008] As used herein, the output latency of an LSD-based ASR is the difference between the time a frame of acoustic features extracted from a speech utterance is received and the time one or more output labels in the received acoustic frame are identified. For example, when an LSD-based ASR system operates on an entire speech utterance, the identification of one or more labels in the utterance is delayed until the last word of the utterance is received. This recognition delay results in increased output latency.

[0009] Some embodiments are based on the recognition that a priori knowledge about the relevance of different parts of the input sequence for identifying the next output label is indicative of the location of the acoustic frames corresponding to the label to be recognized from the input sequence. In fact, if the locations of the acoustic frames that include the relevant label information are known, the LSD-based model can be guided to focus only on these locations rather than all possible locations by restricting the input sequence. In this way, for each output label, the LSD-based model can focus its attention around the location of that label in the input sequence. This guided attention reduces the need to process large input sequences, which in turn reduces output latency, making LSD-based models practical for recognizing speech utterances in a streaming / online manner.

[0010] Some embodiments are based on the recognition that a frame synchronization decoder (FSD) module can be used to provide an indication of the location of the acoustic frame corresponding to the character in the input sequence to the LSD module. This allows the LSD module to limit its attention to the relevant part of the input sequence in order to identify the output label. The FSD-based module generates an output for each frame of the input sequence, that is, the input sequence and the output sequence have the same length. The performance of the LSD-based ASR system can be better than the FSD-based ASR system. However, some embodiments are based on the recognition that the input sequence and output sequence alignment used by the intermediate operations of the FSD-based ASR system can be used by the LSD-based ASR system to solve one or more of the above-mentioned problems.

[0011] Some embodiments are based on the recognition that an FSD module and an LSD module can be jointly trained in a triggered attention (TA) neural network to improve speech recognition for streaming (or "real-time") applications. The TA neural network can include an encoder neural network, an FSD neural network, an LSD neural network, and a synchronization module. The encoder neural network is configured to encode acoustic information included in an acoustic feature frame into a sequence of encoder states. The FSD module is configured to process each encoder state successively at a time and identify an encoder state that updates a prefix list and an associated FSD score list maintained by the FSD module. The FSD module can retain a prefix list and an associated FSD score list generated by processing a previous encoder state. The prefix list and the FSD score list are updated when decoding a subsequent encoder state. For example, in response to identifying an encoder state that carries information about a new transcription output, the FSD module expands a current FSD prefix list to generate a candidate FSD prefix list and a corresponding FSD score maintained by the FSD module.

[0012] The synchronization module is used to provide the prefix list generated by the FSD module to the LSD module. This enables the LSD module to process the same prefix list generated by the FSD module. This enables the FSD module and the LSD module to be synchronized in the prefix domain. In addition, the synchronization module also provides the LSD module with a selected portion of the encoder state, which depends on the encoder state identified by the FSD module that has expanded the current FSD prefix list. This enables the LSD module to focus on the selected portion of the encoder state sequence rather than the entire encoder state sequence corresponding to the entire speech utterance. This reduces the output delay of the LSD module and synchronizes the timing of the FSD module and the LSD module. In this way, the FSD module and the LSD module are synchronized in the time domain and the prefix domain to generate utterance transcriptions for the same sequence of encoder states.

[0013] Some embodiments are based on the recognition that a joint scoring module can generate a joint transcription output list decoded by the FSD module and the LSD module, and further calculate an associated joint score list. The joint scoring module can prune the candidate prefix list based on the joint FSD score and the LSD score. Additionally or alternatively, the joint scoring module can output the prefix with the highest joint score as the transcription output for the current portion of the speech utterance.

[0014] Thus, one embodiment discloses a computer-based automatic speech recognition system, wherein the system uses a processor coupled to stored instructions of a module implementing the system, wherein the processor executing the instructions is configured to: encode an incoming stream of acoustic frames representing speech utterance features into a sequence of encoder states of encoded acoustic feature frames; and continuously process each encoder state of the encoded acoustic feature frames with a frame synchronization decoder (FSD) module until a termination condition is satisfied, wherein the FSD module, in response to identifying an encoder state carrying information about a new transcription output, expands a current FSD prefix list to produce a candidate FSD prefix list and a corresponding FSD score, wherein each FSD prefix in the candidate list is a result of the FSD module's evaluation of a decoded transcription in an encoder state processed by the FSD module. An estimate of an output, wherein a probability that an FSD prefix is a decoded transcription output is defined by a corresponding FSD score, wherein, in response to generating a candidate FSD prefix list, a processor is configured to select a portion of the encoder state that includes the encoder state identified by the FSD module, and trigger a label synchronized decoder (LSD) module to process the selected portion of the encoder state to determine an LSD score according to the LSD module, the LSD score defining a probability that an FSD prefix in the candidate FSD prefix list is a decoded transcription output in the selected portion of the encoder state; prune the candidate FSD prefix list according to a joint score defined by a combination of the corresponding FSD score and the LSD score; and replace the current FSD prefix list with the pruned FSD prefix list to continue decoding a next transcription output.

[0015] Another embodiment discloses a non-transitory computer-readable storage medium having implemented thereon a program including modules executable by a processor to perform automatic speech recognition, the modules including: an encoder configured to encode an incoming stream of acoustic frames representing speech utterance features into a sequence of encoder states of encoded acoustic feature frames; a frame synchronization decoder (FSD) module configured to continuously process each encoder state of the encoded acoustic feature frames to expand a current FSD prefix list in response to identifying an encoder state carrying information about a new transcription output to generate a candidate FSD prefix list and a corresponding FSD score, each FSD prefix in the candidate list being an estimate by the FSD module of a decoded transcription output in an encoder state processed by the FSD module, wherein the FSD prefix is a probability of the decoded transcription output. a synchronization module configured to select a portion of the encoder state including the encoder state identified by the FSD module; a tag-synchronized decoder LSD module configured to process the selected portion of the encoder state to combine the FSD score of the FSD prefix with the corresponding LSD score defining the probability that the FSD prefix is a decoded transcription output according to the LSD module; a joint scoring module configured to prune the FSD prefix according to the joint FSD score and the LSD score and replace the current FSD prefix list with the pruned FSD prefix list to continue decoding the next transcription output; and an output interface configured to output the FSD prefix with the highest joint score as the decoded transcription output of the incoming acoustic frame stream of the encoder state processed by the FSD module.

[0016] Yet another embodiment discloses a method for automatic speech recognition, wherein the method uses a processor coupled to stored instructions for implementing the method, wherein when executed by the processor, the instructions perform the steps of the method, the steps comprising: encoding an incoming stream of acoustic frames representing speech utterance features into a sequence of encoder states of encoded acoustic feature frames; continuously processing each encoder state of the encoded acoustic feature frames with a frame synchronization decoder (FSD) module until a termination condition is satisfied, wherein the FSD module, in response to identifying an encoder state carrying information about a new transcription output, expands a current FSD prefix list to produce a candidate FSD prefix list and a corresponding FSD score, wherein each FSD prefix in the candidate list is a candidate FSD prefix selected by the FSD module for an encoder state selected by the FSD module. an estimate of a decoded transcribed output in a processed encoder state, wherein a probability that an FSD prefix is the decoded transcribed output is defined by a corresponding FSD score; selecting a portion of the encoder state comprising the encoder state identified by the FSD module; triggering a label synchronized decoder (LSD) module to process the selected portion of the encoder state to determine an LSD score according to the LSD module, the LSD score defining a probability that an FSD prefix in a candidate FSD prefix list is the decoded transcribed output in the selected portion of the encoder state; pruning the candidate FSD prefix list according to a joint score defined by a combination of the corresponding FSD score and the LSD score; and replacing a current FSD prefix list with the pruned FSD prefix list to continue decoding a next transcribed output. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A A schematic diagram illustrating an automatic speech recognition (ASR) system configured for end-to-end speech recognition in accordance with some implementations.

[0018] Figure 1B Exemplary prefix lists and associated scores generated by a frame synchronization decoder (FSD) module, a label synchronization decoder (LSD) module, and a joint scoring module are shown.

[0019] Figure 1C A block diagram of an automatic speech recognition method according to some embodiments is shown.

[0020] Figure 1D Exemplary operation of the FSD module according to some embodiments is shown.

[0021] Figure 1E An example of partitioning to limit the future context of an encoder state sequence is shown in accordance with some embodiments.

[0022] Figure 1F An example of partitioning to limit the future context of an encoder state sequence is shown in accordance with some embodiments.

[0023] Figure 2A A block diagram of a triggered attention neural network for an end-to-end speech recognition system according to one embodiment is shown.

[0024] Figure 2B An implementation of an encoder using a deep delay architecture according to an exemplary embodiment is shown.

[0025] Figure 2C A transformer architecture for implementing the LSD module and encoder neural network according to an exemplary embodiment is shown.

[0026] Figure 2D The FSD module is shown to be implemented using a Recurrent Neural Network Transducer (RNN-T) architecture.

[0027] Figure 3A An exemplary architecture for implementing the LSD module as an attention-based decoder with hidden decoder states from previous time steps is shown in accordance with some embodiments.

[0028] Figure 3B An exemplary architecture for implementing the LSD module as an attention-based decoder without hidden decoder states from previous time steps is shown in accordance with some embodiments.

[0029] Figure 4 A block diagram of a computer-based automatic speech recognition system is shown, according to some implementations.

[0030] Figure 5 A data flow diagram of an ASR system according to some embodiments is shown.

[0031] Figure 6 A schematic diagram illustrating an implementation of an FSD module using a prefix beam search module in accordance with some embodiments.

[0032] Figure 7 A block diagram illustrating some components that may be used in various configurations for implementing an ASR system, according to some embodiments. DETAILED DESCRIPTION

[0033] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown in block diagram form only to avoid obscuring the present disclosure.

[0034] As used in this specification and claims, the terms "for example," "for example," and "such as," and the verbs "to include," "to have," "to include," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, are each to be understood as open-ended, meaning that the list should not be construed to exclude other additional components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the phraseology and terminology used herein are for descriptive purposes only and should not be considered limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0035] Figure 1A A schematic diagram of an automatic speech recognition (ASR) system configured for end-to-end speech recognition according to some embodiments is shown. The speech recognition system 100 receives an incoming stream of acoustic frames representing features of a speech utterance and processes the stream of acoustic frames to generate a sequence of transcription outputs. Each transcription output sequence is a transcription of an utterance or a portion of an utterance represented by a corresponding input acoustic signal. For example, the ASR system 100 may receive an incoming stream of acoustic frames 101 and generate a corresponding transcription output 125 that is a transcription of the utterance represented by the incoming stream of acoustic frames 101.

[0036] The incoming acoustic feature frame stream 101 may include a sequence of audio samples that are digital representations of an utterance, e.g., a continuous data stream. Each feature frame extracted from the audio data may correspond to a sequence of time steps, e.g., in which each audio data frame is associated with a 25 millisecond audio sample that is further offset in time by 10 milliseconds from the previous frame of audio data. Each audio data frame in the sequence of feature frames of audio data may include acoustic information that characterizes a portion of the utterance at the corresponding time step. For example, the sequence of feature frames of audio data may include a filter bank spectral energy vector.

[0037] The transcription output or markup output 125 may include a sequence of transcribed segments of the utterance represented by the incoming acoustic frame stream 101. The transcription output 125 may include one or more characters. For example, the transcription output 125 may be a character or sequence of characters from a Unicode character set. For example, the character set may include alphabets for English, Asian, Cyrillic, and Arabic. The character set may also include Arabic numerals, space characters, and punctuation marks. Additionally or alternatively, the transcription output may include byte encodings, words, and other language constructs.

[0038] The ASR system 100 includes an encoder 103, a frame synchronization decoder (FSD) module 107, a synchronization module 109, a label synchronization decoder (LSD) module 111, a joint scoring module 115, and an external language model 122. The present disclosure provides a system for implementing the FSD module 107 and the LSD module 111 in combination to construct an ASR system 100 suitable for "real-time" speech recognition. An end-to-end ASR system with FSD can be easily applied to streaming / online ASR applications where the ASR output must be generated immediately after each spoken word with a small output delay. However, LSD-based ASR systems, which generally demonstrate excellent results in speech recognition, are less suitable for online / streaming ASR because the entire speech utterance is typically required as input due to the loss of alignment information before decoding (i.e., lack of information about whether the input sequence contains enough information to generate the next output token). However, the FSD module 107 and the LSD module 111 can be implemented together in a system 100 to achieve fast and efficient ASR that can be used in streaming ASR applications.

[0039] To this end, the joint scoring module 115 provides a joint score for the transcribed output decoded by the FSD module 107 and the transcribed output decoded by the LSD module 111. However, the FSD module 107 and the LSD module 111 operate based on two different operating principles. In order to implement the FSD module 107 and the LSD module 111 together, they need to be synchronized. Synchronization enables the joint scoring module 115 to provide a joint score for the lists of corresponding prefixes decoded by the FSD module 107 and the LSD module 111. If these modules 107, 111 are not synchronized, the prefix list generated by the FSD module 107 may be different from the prefix list generated by the LSD module 111. In addition, the time at which the modules 107, 111 generate the prefix lists may also be different because the FSD module 107 operates on a frame-by-frame or encoder-state basis, while the LSD module 111 operates on a label-by-label basis and consumes a sequence of input frames or encoder states. Therefore, without synchronization of the modules 107 , 111 in the time and prefix domains, the joint scoring module 115 cannot combine the two transcription hypotheses and generate an intermediate output with low latency.

[0040] For example, the ASR system 100 may receive an incoming stream of acoustic feature frames corresponding to a speech signal for the word "dog." Furthermore, if at time n, the FSD module 107 generates a prefix list that includes: "dog" with an FSD score of 0.3, "dah" with an FSD score of 0.2, and "dag" with an FSD score of 0.1, and at another time, e.g., n+4, the LSD module 111 generates a prefix list that includes: "dug" with an LSD score of 0.3, "dah" with an LSD score of 0.2, and "dag" with an LSD score of 0.5, then, if the modules are not synchronized in the time and prefix domains, the joint scoring module 115 cannot perform joint scoring due to the different prefix lists, and cannot produce a joint output with low latency after each spoken word because it must wait for the LSD module.

[0041] The LSD module 111 is synchronized with the FSD module 107 in the time domain to reduce the output latency of the LSD module. Output latency is introduced in LSD-based ASR systems because LSD-based ASR systems may need to observe the entire speech utterance, segmented by speech pauses, to assign weights to each input frame in order to recognize each transcription output 125. Due to the lack of a priori knowledge about which part of the input acoustic signal is relevant to identifying the next transcription output and the need to assign weights to each encoder state, the LSD module 111 typically needs to process large input sequences. Such processing allows for the advantages of focusing on different parts of the speech, but also increases output latency and is therefore impractical for speech recognition in a streaming / online manner.

[0042] As used herein, the output latency of an ASR system is the time difference between receiving a frame of a speech signal and recognizing the received acoustic information. For example, when an LSD-based ASR system operates on an entire speech utterance, the recognition of the words in the utterance is delayed until the last word of the utterance is received. This recognition delay results in increased output latency.

[0043] The prior knowledge about the relevance of different parts of the input sequence for identifying the next transcribed output is an indication of the position of the frames in the input sequence that correspond to the transcribed output to be recognized. In fact, if the positions of the frames that encode the relevant information for the transcribed output are known, the attention mechanism of the LSD module 111 can be restricted to these positions plus a configurable number of past and future encoder frames or states. In this way, for each transcribed output, the LSD module 111 can focus its attention around these positions in the input sequence. This guided attention reduces the need to process large input sequences, which in turn reduces output latency, making the LSD module 111 practical for speech recognition in a streaming / online manner.

[0044] To this end, the ASR 100 uses an FSD module 107, which acts as both an alignment decoder and a transcription output decoder. The FSD module 107 is trained to determine the position 119 of the encoder state in the sequence 105 that encodes the transcription output 125 (such as a character, byte encoding, word, etc.). In some embodiments, the FSD module 107 can be implemented using a connectionist temporal classification (CTC) neural network objective. Additionally, in some embodiments, the LSD module 111 can be implemented using an attention-based decoder. CTC is an objective function used to train neural networks such as long short-term memory (LSTM) neural networks and associate neural network outputs to solve time-variable sequence problems. FSD-based ASR systems are an alternative to LSD-based ASR systems. FSD generates an output for each frame of the input sequence, i.e., the input and output are synchronized, and a beam search algorithm is used to find the optimal output sequence before folding the neural network output into a prefix and outputting the transcription. LSD-based ASR systems can outperform FSD-based ASR systems. However, some embodiments are based on the recognition that the input and output frame alignment used by the intermediate operations of the FSD-based ASR system can be used by the LSD-based ASR system to address its above-mentioned output delay disadvantage.

[0045] To utilize the alignment information 119 provided by the FSD module 107, the ASR system 100 includes a synchronization module 109 that is configured to partition the encoder state sequence 105 into a set of partitions 121. For example, the synchronization module 109 may partition the encoder state sequence for each position 119 of the identified encoder state such that the number of partitions 121 is defined by (e.g., is equal to) the number of identified encoder states 119 that are encoded for the transcription output 125.

[0046] The synchronization module also submits the current prefix list and a selected portion of the encoder state sequence generated by the FSD module 107 to the LSD module 111. This ensures that the LSD module 111 processes the same prefix list as the FSD module 107 in order to estimate the likelihood of the LSD module's transcription output. Thus, synchronization between the FSD module 107 and the LSD module 111 is achieved in the prefix domain. Furthermore, to ensure that the LSD module 111 outputs the LSD score list for the prefix list at approximately the same time as the FSD module 107 outputs the FSD score list, the synchronization module 109 only provides the LSD module 111 with the selected portion of the encoder state sequence 105. This enables the LSD module 111 to focus on the selected portion of the encoder state sequence 105 rather than the entire sequence 105, which reduces the output latency of the LSD module 111. Thus, synchronization between the FSD module 107 and the LSD module 111 is achieved in the time domain.

[0047] The ASR system 100 includes an encoder 103 that processes an incoming stream of acoustic feature frames 101 and generates an encoder state sequence 105 that provides an alternative (e.g., higher) representation of the input acoustic signal 101. The encoder state sequence 105 may include an alternative sequence of feature frames of audio data corresponding to a second set of time steps. In some implementations, the alternative representation of the input acoustic sequence is subsampled to a lower frame rate, i.e., the second set of time steps in the alternative representation are smaller than the first set of time steps in the input acoustic sequence 101.

[0048] In addition, the FSD module 107 is configured (e.g., trained) to continuously process the encoder state 105 and update the prefix list and associated FSD score list maintained by the FSD module 107. The FSD module 107 is configured to continuously process each encoder state of the encoded feature to identify an encoder state that updates the FSD prefix list maintained by the FSD module. In this way, the FSD module expands the current FSD prefix list in response to identifying an encoder state that carries information about the new transcription output to produce a candidate FSD prefix list. Each FSD prefix is a candidate estimate of the decoded transcription output by the FSD module, and its probability is defined by its corresponding FSD score, that is, the FSD score defines the probability that the FSD prefix is the decoded transcription output. The FSD prefix list and the FSD score list are updated when decoding subsequent encoder states. For example, the current FSD prefix list determined during the previous iteration is expanded to the candidate prefix list for the current iteration. The FSD module 107 provides (117) the candidate FSD prefix list and the associated FSD score list to the joint scoring module 115. The joint scoring module 115 prunes the candidate FSD prefix list based on the LSD and FSD scores to form the current FSD prefix list for the next iteration. In this way, the computational efficiency of the FSD module is maintained without sacrificing decoding quality.

[0049] To achieve this result, in some embodiments, the synchronization module 109 is configured to select a portion of the encoder state sequence, wherein the portion includes the encoder state identified by the FSD module 107 in response to an update of a prefix list maintained by the FSD module 107. The synchronization module 107 provides the selected portion of the encoder state sequence and the corresponding prefixes to the LSD module 111. The LSD module 111 is configured (e.g., trained) to process the selected portion of the encoder state sequence to update the LSD prefix list and the corresponding LSD score maintained by the LSD module 111. Similarly, for the LSD module, each LSD prefix is a candidate estimate of the decoded transcription output by the LSD module, whose probability is defined by its corresponding LSD score. In some embodiments, the synchronization module 109 passes the FSD prefixes to the LSD module, and the LSD module only determines the LSD score for the FSD prefixes. In this way, the FSD and LSD modules are synchronized in the prefix domain.

[0050] Decoding only a selected portion of the encoder state sequence enables the LSD module 111 to focus only on the selected portion of the sequence 105 rather than the entire sequence 105, which reduces the output delay of the LSD module 111. In this way, synchronization of the FSD module 107 and the LSD module 111 in the time domain and the prefix domain can be achieved. Therefore, the FSD module 107 and the LSD module 111 decode the same encoder state sequence at approximately the same time.

[0051] Furthermore, to improve the accuracy of the transcription output, the ASR system 100 includes a joint scoring module 115. The joint scoring module 115 combines the FSD scores and LSD scores of corresponding prefixes in the updated prefix lists maintained by the FSD module 107 and the LSD module 111 to generate a joint prefix list and an associated joint score list. Each joint score in the joint score list is a weighted combination of the FSD score of the corresponding prefix in the updated prefix list in the FSD module 107 and the LSD score of the corresponding prefix in the updated prefix list in the LSD module 111. In addition, the score provided by the external language model 122 can be weighted and added to the joint FSD and LSD scores to further improve recognition accuracy.

[0052] In addition, the joint scoring module 115 feeds back (117) the pruned joint prefix list to the FSD module 107, where the pruning stage removes unlikely prefixes from the prefix list based on the corresponding joint scores. Thus, the joint scoring module 115 prunes the candidate FSD prefix list according to a joint score defined by a combination of the corresponding FSD score and the LSD score. The pruning from the joint scoring module 115 synchronizes the selection of the best prefixes by the FSD module 107 and the LSD module 111 for subsequent decoding of the encoder states at subsequent time steps. This further increases the accuracy of the transcription output 125 decoded from the subsequent encoder states by the FSD module 107 and the LSD module 111. In addition, the joint scoring module 115 can output the prefix with the highest joint score as the current portion of the transcription of the incoming acoustic frame stream 101 for the current time step.

[0053] In some embodiments, the FSD module may use an external language model to update and pre-prune the generated list of FSD prefixes before communicating with the synchronization module and before triggering the LSD module in order to improve recognition accuracy and reduce the computational load incurred by the LSD module due to pre-pruning.

[0054] In some embodiments, the combination of the encoder 103, the FSD module 107, the synchronization module 109, and the LSD module 111 is referred to as a triggered attention (TA) model or a TA neural network. In practice, the TA model can process a partial utterance as it is received, making the ASR system 100 practical for recognition in a streaming / online manner. Figure 1A exist Figure 1B The workings of the FSD module 107, the LSD module 111, and the joint scoring module 115 with an exemplary prefix list are explained in FIG.

[0055] Figure 1B Exemplary prefix lists and associated scores generated by the FSD module, the LSD module, and the joint scoring module are shown. Table 191 shows the prefix list generated by the FSD module, Table 192 shows the prefix list generated by the LSD module, and Table 193 shows the prefix list generated by the joint scoring module.

[0056] The joint scoring module 115 combines the matching FSD prefixes and LSD prefixes and their corresponding FSD scores 191 and LSD scores 192 to generate a joint prefix list and a corresponding joint score list 193. For example, in one embodiment, the joint scoring module 115 processes the attention decoder transcription output score 112 (LSD score) and the FSD module output score 117 (FSD score) to jointly score the two transcription outputs and find the best joint transcription output sequence for the utterance. For example, for each time step, the joint scoring module 115 can output the best joint score transcription, which can be calculated by the weighted sum of the two transcription output scores as shown in the example with a weighting factor of 0.5. In addition to calculating the joint FSD and LSD scores, the scores of the external language model can also be weighted and added.

[0057] Next, the joint scoring module 115 replaces 194 the FSD prefix list maintained by the FSD module with a pruned joint prefix list, which is a reduced list of the best joint prefixes. This replacement 194 synchronizes the FSD and LSD modules in the prefix domain by selecting the best joint prefix hypothesis for further processing and removing unlikely prefix candidates based on the joint score. The LSD module 111 receives the current active prefix list from the FSD module 107 via the synchronization module 109.

[0058] For example, the input acoustic feature frame may correspond to the word "dog". The FSD module 107 receives the encoder state sequence 105 of the encoded acoustic feature frame 101. As shown in Table 191, the FSD module 107 generates an FSD prefix list with a corresponding FSD score. Each prefix in the prefix list is a candidate estimate of the transcription output 125 corresponding to the received acoustic frame. The FSD module 107 predicts the transcription output as "dog" and "dah" with the highest probability (i.e., FSD score 0.3), and predicts the transcription output as "duh" with the lowest probability 0.1. In addition, the FSD module 107 provides the synchronization module 109 with the position of the encoder state that encodes the transcription output 125 and the FSD prefix list.

[0059] The synchronization module 109 selects a portion of the encoder state sequence. The selected portion includes the encoder state identified by the FSD module 107. The synchronization module 109 provides the selected portion of the encoder state sequence and the corresponding prefix to the LSD module 111.

[0060] In addition, the LSD module 111 decodes the portion of the encoder state sequence to generate an LSD prefix list and an associated probability list, i.e., an LSD score, as shown in Table 192. Unlike the FSD module 107, the LSD module 111 predicts the transcription output as "dog" with the highest probability of 0.4, and predicts the transcription outputs as "duh" and "dah" with lower probabilities. The probability scores generated by both the FSD module 107 and the LSD module 111 can be combined to accurately predict the transcription output "dog".

[0061] To this end, the joint scoring module 115 obtains a list of prefixes and associated scores from both the FSD module 107 and the LSD module 111. To determine the most accurate output, the joint scoring module 115 may combine the FSD scores and LSD scores of the corresponding prefixes. The combination of scores may correspond to a weighted average. However, any other mathematical operation / equation may be used to achieve the combination of scores. Thus, as shown in Table 193, the joint scoring module 115 generates a joint prefix list and associated joint probabilities, where the prefix "dog" corresponds to the highest joint probability. Therefore, the joint scoring module 115 outputs "dog" as the transcription output for the input acoustic frame at the current time step.

[0062] Figure 1C A block diagram of an automatic speech recognition method according to some embodiments is shown. The embodiment encodes (180) an incoming stream of acoustic frames representing features of a speech utterance into a sequence of encoder states of encoded acoustic feature frames, and continuously processes (181) each encoder state of the encoded acoustic feature frames using an FSD module until a termination condition is satisfied, for example, until the speech utterance ends. The FSD module, in response to identifying an encoder state that carries information about a new transcription output, expands a current FSD prefix list 189 to produce a candidate FSD prefix list 186 and a corresponding FSD score 185 for the candidate FSD prefix list 186.

[0063] by Figure 1BFor example, let the current FSD prefix list 189 include the prefixes "do," "du," and "da." In response to identifying an encoder state that carries information about the newly transcribed output, the FSD module expands the current FSD prefix lists of "do," "du," and "da" to generate a candidate FSD prefix list 186 of "dog," "duh," and "dah." In practice, the candidate FSD prefix list will include many prefixes that need to be pruned to maintain computational efficiency. For example, in some implementations, the size of the current FSD prefix list is fixed and smaller than the size of the candidate FSD prefix list, such that pruning retains the n best FSD prefixes in the candidate FSD prefix list with the highest joint scores, where n is the size of the current FSD prefix list. To this end, the joint scoring module needs to prune the candidate FSD prefix list based on the FSD scores. In this example, the "duh" FSD prefix needs to be pruned because it has the lowest FSD score. However, some embodiments use both the FSD score and the LSD score to prune the candidate FSD prefix list based on the joint score, rather than solely on the FSD score, which improves pruning accuracy.

[0064] To this end, some embodiments trigger (182) a label synchronized decoder (LSD) module to process the portion of the encoder state sequence processed by the FSD module to determine an LSD score 188 based on the LSD module, the LSD score defining the probability that an FSD prefix in the candidate FSD prefix list is a decoded transcribed output in the portion of the encoder state sequence. Such triggering, for example, implemented using a synchronization module, synchronizes the FSD module and the LSD module in the time domain and the prefix domain as the LSD module processes the portion of the encoder state including the new transcribed output identified by the FSD module and estimates the LSD score 188 for the FSD prefix 186.

[0065] Using the LSD scores and the FSD scores for the FSD prefixes 186, some embodiments prune (183) the candidate FSD prefix list according to a joint score defined by a combination of the corresponding FSD scores and the LSD scores, and replace (184) the current FSD prefix list 189 with the pruned FSD prefix list 187 for use in decoding the next transcription output. For example, the FSD prefix "duh" may be pruned due to a minimum joint score of 0.2.

[0066] One embodiment replaces only the FSD prefixes without replacing the FSD scores in the replaced current FSD prefix list. For example, after pruning the FSD prefix "duh", the current list will have the prefixes "dog" and "dah", whose original FSD scores are 0.3 and 0.3 respectively. In this way, the continuity of FSD decoding is maintained.

[0067] In various embodiments, the decoding process is performed iteratively, so that different locations of the transcription output of "d", "o", "a", "g", and / or "h" are identified for different iterations. In some implementations, the FSD module, the LSD module, and the joint scoring module iteratively decode the transcription of the incoming acoustic frame stream such that for each iteration, the prefixes in the FSD and / or LSD prefix lists are updated with new transcription output tokens.

[0068] Figure 1D An exemplary operation of the FSD module is shown. One of the purposes of the FSD module 107 is to decode the encoder state sequence 105 generated by the encoder 103. To this end, the FSD module 107 is trained to decode the sequence 105 to produce a transcribed output sequence 125. The FSD module 107 processes the encoder state sequence 105 frame by frame and computes a prefix sequence or prefix list corresponding to the intermediate transcribed output. Furthermore, the FSD module 107 generates an FSD score list associated with the prefix list. The FSD score list can be generated by marginalizing the probabilities of all paths of the FSD neural network that lead to the same prefix and intermediate transcribed output. Extending the prefix sequence by appending newly recognized characters provides alignment information for both time and frame positions. In some embodiments, the alignment information generated by the FSD module 107 is used to partition the encoder state sequence and trigger an attention-based decoding process implemented using the LSD module 111. The underlying principle behind this approach is that the prefix scores of the LSD module 111 can be better and more reliable than those of the FSD module 107. To this end, in the absence of alignment information, the intermediate alignment information generated by the FSD module 111 is used to further improve the processing of the LSD module 111 by reducing the output delay generated by the LSD module 111.

[0069] In some embodiments, the joint scoring module 115 is implemented to generate a joint score for the transcription output decoded by the FSD module 107 and the transcription output decoded by the LSD module 111. This improves the accuracy of the speech recognition of the ASR system. In some embodiments, the FSD module 107 is used twice: first, to help divide the encoder state sequence of the LSD module 111; and second, to further improve the accuracy of the transcription output decoded by the LSD module 111.

[0070] In addition, the box surrounding the index sequence element identifies the position of the encoder state 119 in the encoder state sequence 105 that encodes the transcription output 125. For example, the encoder 103 converts an input acoustic sequence X of acoustic features such as log-Mel spectral energies into an encoder state sequence X of length N. E :

[0071] X E =Encoder(X).

[0072] For example, in one implementation, the encoder output is subsampled to a frame rate four times lower than the feature matrix X having a sampling rate of 100 Hz. Let Z = (z1, ..., z N ) represents a frame-by-frame label sequence of length N, in, denotes different sets of transcription outputs, e.g., word fragments or words that can be single characters, and ∈ is a blank symbol. Let C = (c1, ..., c L ),in, It represents a sequence of tags of length L such that when repeated tags are collapsed into single occurrences and whitespace symbols are removed, the sequence Z reduces to C.

[0073] In some implementations, the FSD module 107 probabilistically decodes the encoder state sequence 105, where the probability is derived as:

[0074]

[0075] Among them, p(Z|C) represents the transformation model, p(Z|X E ) represents the acoustic model.

[0076] In some embodiments, a forced alignment process is used during training to determine the marker sequence Z that is the highest probability marker sequence or CTC path. * The alignment of the subsequences used to condition the LSD module to the encoder state sequence is determined by * The frame with the highest probability within each subsequence of frames corresponding to the same label in is identified as the identified encoder state. For example, consider the sequence Z * is to use index i l and j l These indices are used for the lth label c l In Z * The first and last appearances in And for all t, z n =c l , such that for all other indices, i l ≤n≤j l And z t =∈. TA alignment is performed from sequence Z * to a sequence Z′=(∈ N) of the same length N that includes the subset of identified encoder states that encode the transcription output 125 with the highest probability * ,c1,∈ * ,c2,∈ * ,...,c L ,∈ * ), where *represents zero or more repetitions, and where each c l In the corresponding c l appears once with the maximum probability in the frame, as follows:

[0077]

[0078] Alternatively, the FSD module 107 may * The first or last frame within each subsequence of frames with the same grapheme in is identified as the identified encoder state.

[0079] Figure 1E and Figure 1F An example of partitioning an encoder state sequence according to some embodiments is shown. In various embodiments, the partitioning is performed by a synchronization module 109 operatively connected to the FSD module 107, the LSD module 111, and the encoder 103. The synchronization module 109 is configured to access alignment information 119 of the FSD module 107, partition the encoder state sequence 105 generated by the encoder 103, and sequentially submit portions 121 of the encoder state sequence to the LSD module 111.

[0080] For example, in Figure 1E In one embodiment, each partition 121a includes encoder states from the beginning of the sequence of encoder states 105 up to a leading encoder state determined by moving the position of the identified encoder state forward by a fixed shift. Figure 1A An example of a look-ahead encoder state 123 is shown in . For example, if the value of the fixed shift is 5 and the position of the identified state encoder is the 8th in the encoder state sequence 105, then partition 121a includes the first 13 encoder states. If the position of the subsequently identified encoder state is 11, then partition 121a includes the first 16 encoder states. In effect, each partition includes encoder states for a newly transcribed output while increasing the length of the partition, allowing the LSD module 111 to take advantage of its length.

[0081] exist Figure 1F In an alternative embodiment, the partition 121b corresponding to the position of the identified encoder state includes a predetermined number of look-ahead encoder states and a predetermined number of look-back encoder states relative to the position of the identified encoder state. For example, if the position of the identified encoder state is the 15th in the encoder state sequence, and if the fixed look-ahead and look-back values are 4 and 8, respectively, then the partition 121b includes encoder states between the 7th and 19th in the encoder state sequence 105. In effect, each partition includes encoder states for a newly transcribed output, while having a fixed length partition reduces the computational burden on the LSD module 111.

[0082] Figure 2A A block diagram of a triggered attention neural network 200 of an end-to-end speech recognition system according to one embodiment is shown. In this embodiment, the encoder 103, the FSD module 107, and the LSD module 111 are implemented as neural networks. Furthermore, the FSD module 107 is implemented using a connectionist temporal classification (CTC)-based neural network or CTC module, and the LSD module 111 is implemented using an attention-based decoder neural network. The triggered attention model 200 includes an encoder network module 203, encoder network parameters 205, an attention decoder network module 206, decoder network parameters 207, a synchronization module 209, a CTC module 211, and CTC network parameters 213. The encoder network parameters 205, the decoder network parameters 207, and the CTC network parameters 213 are stored in a storage device to provide parameters to the corresponding modules 203, 206, and 211. An acoustic feature sequence 201 is extracted from the audio waveform data and can be stored in a storage device and provided to the encoder network module 203. The audio waveform data may be obtained via an input device using a digital signal processing module (not shown) that receives and processes speech sounds in the audio data.

[0083] The encoder network module 203 includes an encoder network that converts the acoustic feature sequence 201 into an encoder feature vector sequence using an encoder network whose parameters are read from the encoder network parameters 203. The CTC module 211 receives the embedding vector sequence from the encoder network module 203 and uses the CTC network parameters 213 and dynamic programming techniques to calculate the CTC-based posterior probability distribution of the label sequence. After calculation, the CTC module 211 provides the most likely label sequence or prefix including the frame position to the synchronization module 209.

[0084] The attention decoder network module 206 includes a decoder network. The attention decoder network module 206 receives each partition including a portion of the encoder state sequence from the synchronization module 209, and then uses the decoder network whose parameters are read from the decoder network parameters 205 to calculate the attention-based posterior probability distribution of the label.

[0085] End-to-end speech recognition is usually defined as finding the most likely label sequence given an input acoustic feature sequence X. The problem is,

[0086]

[0087] in, Given a set of predefined labels A set of possible tag sequences.

[0088] In end-to-end speech recognition, p(Y|X) is computed by a pre-trained neural network, which does not require a phonetic dictionary and does not require a graph search based on a weighted finite state transducer (WFST). In the state-of-the-art attention-based end-to-end speech recognition, the neural network consists of an encoder network and a decoder network.

[0089] Encoder Example

[0090] The encoder network module 203 includes a module for converting the acoustic feature sequence X=x1, ..., x T Convert to a sequence of embedding vectors The encoder network, such as

[0091] X E =Encoder(X), (2)

[0092] Among them, the function Encoder(X) may include one or more recurrent neural networks (RNNs), convolutional neural networks (CNNs), feedforward neural networks, or self-attention neural networks stacked into a deep architecture. The RNN can be implemented as a long short-term memory (LSTM), which has an input gate, a forget gate, an output gate, and a storage unit in each hidden unit. The other RNN can be a bidirectional RNN (BRNN) or a bidirectional LSTM (BLSTM). BLSTM is a pair of LSTM RNNs, one is a forward LSTM and the other is a backward LSTM. The embedding vector of the BLSTM is obtained as a cascade of the hidden vectors of the forward LSTM and the backward LSTM.

[0093] For the forward LSTM, the forward n-th hidden vector Calculated as

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] where σ(·) is the element-wise sigmoid function, tanh(·) is the element-wise hyperbolic tangent function, and They are x t The input gate, forget gate, output gate and unit activation vector of . ⊙ represents the element-by-element multiplication between vectors. Weight matrix and the bias vector are the parameters of LSTM, which are identified by subscript z∈{x,h,i,f,o,c}. For example, is the hidden to input gate matrix, is the input to output gate matrix. The hidden vector is the input vector x t and the previous hidden vector Recursively obtained, where The wave is assumed to be a zero vector.

[0100] Using backward LSTM, backward t-th hidden vector Calculated as:

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] in, and They are x t The input gate, forget gate, output gate and unit activation vector of . Weight matrix and the bias vector are the parameters of the LSTM, which are identified by subscripts in the same way as the forward LSTM. The hidden vector is the input vector x t and the subsequent hidden vector Recursively obtained, where is assumed to be the zero vector.

[0107] The hidden vector of BLSTM is obtained by concatenating the forward hidden vector and the backward hidden vector as follows:

[0108] Here, T represents the transpose operation of the vectors assuming that all vectors are column vectors. and are considered as parameters of BLSTM.

[0109] To obtain better hidden vectors, some implementations stack multiple BLSTMs by feeding the hidden vector of the first BLSTM to the second BLSTM, and then feeding the hidden vector of the second BLSTM to the third BLSTM, and so on. t′ is the hidden vector obtained by a BLSTM, then when it is fed to another BLSTM, x t =h t ’. To reduce computation, some embodiments may feed only every second hidden vector of a BLSTM to another BLSTM. In this case, the length of the output hidden vector sequence becomes half the length of the input acoustic feature sequence.

[0110] In an exemplary embodiment, the encoder module is implemented using a time-delayed long short-term memory (TDLSTM) neural network and a parallel time-delayed LSTM (PTDLSTM) neural network constructed together using a deep time-delay architecture. Figure 2B The implementation of the encoder module using a deep time delay architecture is shown in Figure 2B In Figure 1, the encoder architecture using a deep time delay structure is shown on the left, and the neural network building blocks are shown on the right. Each rectangular box in the deep time delay structure represents a TDLSTM building block in layer 1 or a PTDLSTM building block in layers 2-5. The numbers in square brackets indicate the frame-delayed input to each layer. The solid black lines and shaded rectangular boxes highlight the path of a single encoder output frame. The dashed lines and dark rectangular boxes represent the connections and building blocks used to generate past and future encoder output frames.

[0111] In this example, the deep time delay architecture generates a delay of 25 input acoustic frames, the sum of which corresponds to 250ms. The first neural network layer (layer-1) includes a TDLSTM building block that takes three consecutive acoustic frames as input and thereby generates an output at a frame rate three times lower, i.e., applying a subsampling factor of 3. The remaining encoder neural network layers, i.e., layers 2-5, are based on PTDLSTM. Each of the TDLSTM and PTDLSTM neural network architectures includes multiple layers of neural network blocks. In addition, each of the multiple layers can be composed of different neural network building blocks. The architectures of TDLSTM and PTDLSTM are explained below.

[0112] The different neural network building blocks included in the TDLSTM are: input concatenation (cat) block 215, LSTM layer (LSTM) block 217, bottleneck (BN) feedforward neural network block 219, and rectified linear unit activation function (ReLU) block 221. The input concatenation block 215 first concatenates all input acoustic frames and provides the concatenated input acoustic frames to the LSTM block 217. The LSTM block 217 processes the time delay and concatenated inputs before the bottleneck feedforward neural network 219. The bottleneck feedforward neural network block 219 reduces the size of the input acoustic frame to the subsequent layer (i.e., the ReLU block 221). The ReLU is an activation function that produces the final encoder state that encodes the transcribed output of the input acoustic frame received from the bottleneck feedforward neural network block 219.

[0113] In addition, the PTDLSTM network architecture includes an LSTM layer 223, a concatenation block 225, a bottleneck feedforward neural network block 227, and a rectified linear unit activation function (ReLU) block 229. In the PTDLSTM network, each time-delayed input acoustic frame is processed by a separate LSTM in the LSTM layer 223, and the LSTM output is provided to the concatenation block 225. The concatenation block 225 concatenates the LSTM outputs before forwarding them to the bottleneck feedforward neural network block 227 and the rectified linear unit (ReLU) activation function 229 to produce a final encoder state that encodes the transcription output of the input acoustic frame.

[0114] Compared to other RNN-based streaming encoder architectures (such as delay-controlled BLSTM) that apply block processing and limit the backward LSTM to a limited future context, the TDLSTM encoder architecture and the PTDLSTM architecture improve the error rate and inference speed of ASR systems. For example, these architectures limit the processing delay to a fixed amount and allow for streaming ASR. In addition, the PTDLSTM architecture demonstrates advantages in word error rate compared to other RNN-based streaming architectures (such as delay-controlled BLSTM architecture).

[0115] LSD module example

[0116] In some embodiments, the LSD module is implemented using an attention-based decoder. The attention-based decoder network module 206 includes a decoder for using the embedding vector sequence X E To calculate the decoder network of the label sequence probability p(Y|X). Assume that Y is a label sequence y1, y2, ..., y L To efficiently compute p(Y|X), the probability can be decomposed using the probability chain rule as:

[0117]

[0118] Each label probability p(y l y1,…,y l-1 , X) is obtained from the probability distribution over labels, which is estimated using the decoder network as:

[0119] p(y|y1,…,y l-1 ,X)=Decoder(r l ,q l-1 ), (15)

[0120] Where y is a set of integers representing the output of each label, r l It is called the content vector, which has X E Content information. l-1 is the decoder state vector, which contains the previous labels y1, ..., y l-1 and the previous content vector r0,...,r l-1 Therefore, the label probability is obtained as y = y given the context l The probability of

[0121] p(y l |y1,…,y l-1 , X) = p(y = y l |y1,…,y l-1 ,X)· (16)

[0122] Content vector r l It is usually given as a weighted sum of the embedding vectors of the encoder network, i.e.,

[0123]

[0124] Among them, a ln It is called satisfying ∑ n a ln =1 attention weight. The attention weight can be calculated using q l-1 and X E Calculated as

[0125]

[0126] f l =F*a l-1 (19)

[0127]

[0128] Where W, V, F, and U are matrices, and w and b are vectors, which are the trainable parameters of the decoder network. ln is the (l-1)th state vector q l-1 and the tth hidden vector The matching scores between them are used to form the time alignment distribution a l ={a ln |n=1,...,N}. l-1 Represents the function used to predict the previous label y l-1 The previous alignment distribution {a (l-1)n |n=1,...,N}. f l ={f ln |t=1,...,N} is for a l-1 The convolution result with F, which is used to reflect the previous alignment to the current alignment. "*" indicates the convolution operation.

[0129] Using the state vector q l-1 and content vector r l The label probability distribution is obtained as

[0130] Decoder(r l ,q l-1 )=softmax(W qy q l-1 +W ry r l +b y ), (twenty one)

[0131] Among them, W qy and W ry is a matrix, and b y are vectors that are trainable parameters of the decoder network. The softmax() function is calculated as

[0132]

[0133] For a K-dimensional vector v, v[i] represents the i-th element of v.

[0134] Afterwards, the decoder state vector q is transformed into l-1 Updated to q l

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] in, and They are the input vector x lThe input gate, forget gate, output gate and unit activation vector of . Weight matrix and the bias vector are the parameters of the LSTM, which are identified by subscripts in the same way as the forward LSTM. and the previous state vector q l-1 Recursively obtain the state vector q l , where q0 is calculated assuming q-1 = 0, y0 = <sos> and a0 = 1 / T. For the decoder network, the input vector is given as label y l and content vector r l The concatenation vector of Here, Embed(·) represents label embedding, which converts the label into a fixed-dimensional vector.

[0141] In attention-based speech recognition, estimating appropriate attention weights is very important for predicting the correct label, since the content vector r l Very dependent on the alignment distribution a l , as shown in formula (17). In speech recognition, the content vector represents the acoustic information in the hidden vector of the encoder near the peak of the alignment distribution, and this acoustic information is the predicted label y l However, the attention mechanism usually provides irregular alignment distributions because there is no explicit constraint such that when incrementally predicting y l When , the peak of the distribution advances monotonically along time. In speech recognition, the alignment of the input sequence and the output sequence should generally be monotonic. Although the convolution feature f lt This mitigates the generation of irregular alignments, but does not guarantee that they will be avoided.

[0142] In another embodiment, the LSD module and the encoder module are implemented using a converter architecture. Figure 2C The transformer architecture implementing the LSD module and encoder neural network is shown. Figure 2CThe transformer architecture shown is an alternative to RNN-based encoders and LSD architectures. The encoder 203 receives a sequence of input frames, where each frame of the sequence includes features associated with the speech input. These features can be extracted by a feature extraction module 231. The encoder 203 neural network includes E stacked layers with the same structure, where each layer has two sub-layers: the first is a multi-head self-attention mechanism 235, and the second is a feedforward neural network layer 239. For better optimization, the encoder 203 uses residual connections, followed by layer normalization 237, 241 for each sub-layer. In addition, a positional encoding 233 is added to the input of the encoder 203, which ensures that the multi-head self-attention 235 can identify the position of the frame within the frame sequence. In some embodiments, the multi-head self-attention mechanism uses restricted self-attention in order to control the latency of the encoder neural network and enable streaming applications.

[0143] In addition, the decoder 205 also includes D stacked layers, each of which has a multi-headed self-attention layer 247 to process the output of the previous decoder block provided by the output embedding 243 (where the first decoder block looks at the output of the last decoder block from the previous processing step). In addition, relative or absolute frame position information in the output embedding sequence calculated from the previous decoding step is provided to the decoder using position encoding 245 at the input of the decoder 205. The decoder 205 includes a multi-headed encoder-decoder attention layer 251 to process the output of the encoder neural network 203 using the query vector generated by the multi-headed self-attention layer 247. The output of the encoder-decoder attention layer 251 is fed into a feedforward neural network layer 255. In addition, similar to the encoder 203, the decoder 205 uses residual connections, followed by layer normalization 249, 253, 257 for each sub-layer to improve generalization and regularization. A learned linear transformation 259 and a softmax function 261 are used to convert the decoder output into predicted label output probabilities.

[0144] FSD module example

[0145] In one embodiment, the FSD module is implemented using the CTC neural network described above. The CTC module 211 calculates the given embedding vector sequence X E The CTC forward probability of the label sequence Y. Note that the CTC formula uses the label sequence Y of length L = (y I ,...,y L ),in, and are different sets of labels. By introducing The frame-by-frame label sequence Z=(z I ,...,z N), where ∈ represents an additional blank label, and using the probability chain rule and conditional independence assumption, the posterior distribution p(Y|X) is decomposed as follows:

[0146]

[0147] Among them, p(z n |z n-I ,Y) is considered as the label transition probability including the blank label. p(z n |X) is the frame-by-frame posterior distribution conditioned on the input sequence X and is modeled by using an encoder neural network as described above followed by a linear transformation and a softmax function:

[0148]

[0149] in, is obtained using an encoder network. is the weight matrix, are the bias vectors, which are the trainable parameters of the CTC model. Although Eq. (28) must deal with summing over all possible Z, it can be computed efficiently using the forward-backward algorithm and dynamic programming.

[0150] For example, the forward algorithm of CTC is performed as follows. Some implementations use an extended label sequence Y′=y′1,y′2,...,y′ of length 2L+1 2L+1 =∈,y1,∈,y2,...,∈,y L ,∈·, where a blank label∈ is inserted between each pair of adjacent labels. Let α t (s) is the forward probability, which represents the label sequence y1,...,y for time frame I,...,t l The posterior probability of , where s indicates the position in the extended label sequence Y ′.

[0151] For initialization, set

[0152] α1(1)=p(z1=∈|X) (30)

[0153] α1(2)=p(z1=y1|X) (31)

[0154]

[0155] When t=2 to T, α t (s) is recursively calculated as

[0156]

[0157] in,

[0158]

[0159] Finally, the CTC-based label sequence probability is obtained as

[0160] p(Y|X)=α T (2L+1)+α T (2L). (35)

[0161] The frame-by-frame label sequence Z represents the alignment between the input acoustic feature sequence X and the output label sequence Y. When computing the forward probability, the recursion of Eq. (33) forces Z to be monotonic and does not allow s to loop or make large jumps in the alignment Z. This is because finding α t The recursion of (s) only considers α at most t-1 (s), α t-1 (s-1), α t-1 (s-2). This means that when the time frame advances by one frame, the label changes from the previous label or blank, or remains the same label. This constraint gives the transition probability p(z t |z t-1 , Y), the transition probability is forced to align monotonically. Therefore, when calculated based on irregular (non-monotonic) alignment, p(Y|X) can be 0 or a very small value. The synchronization module 209 uses the alignment between the input acoustic feature sequence X and the output label sequence Y to control the operation of the attention-based neural network 206.

[0162] In one embodiment, the FSD module may be implemented using a recurrent neural network (RNN) transducer architecture. Figure 2D The implementation of the FSD module based on the Recurrent Neural Network Transducer (RNN-T) architecture is shown. RNN-T is an extension of Connectionist Temporal Classification (CTC). RNN-T includes an encoder 263, a prediction network 265, a joint network 267, and a softmax function 269. The encoder 263 takes as input an acoustic frame sequence X = (x0, ..., x T ) is transformed into a high-level feature representation h enc .

[0163] h enc =Encoder(X) (36)

[0164] The prediction network can eliminate the limitation of the frame-independent assumption in the CTC architecture. The prediction network can use the long short-term memory (LSTM) neural network to model the context information, thereby transforming the original vector y = (y1, ..., y l-1 ) is converted to a high-level representation The output of the prediction network is determined by the previous context information. Note that the first input of the prediction network is an all-zero tensor and y is a non-blank unit. Equations (37) and (38) describe how the prediction network operates at the labeling step l. The joint network is usually a feed-forward network that starts from h n enc and produce

[0165]

[0166] Finally, the posterior for each output token k is calculated by applying the softmax operation

[0167]

[0168] Train the entire network by optimizing the RNN-T loss function

[0169] Loss rnnt = -ln(P(Y|X)), (39)

[0170] It is calculated by the forward-backward algorithm.

[0171] For RNN-T decoding, the most likely character sequence is generated using a beam search algorithm. During inference, the input to the prediction network is the last non-whitespace symbol. The final output sequence is obtained by removing all whitespace symbols in the most likely sequence path.

[0172] Some embodiments are based on the recognition that by combining the decoder output from CTC and the attention-based decoder, recognition accuracy can be further improved. For example, in one implementation of end-to-end speech recognition, the CTC forward probability in Equation (34) is combined with the attention-based probability in Equation (14) to obtain a more accurate label sequence probability. The transducer architecture includes a prediction network 251, an encoder 253, a joint network 255, and a softmax function 257, where the encoder network can be shared with the attention-based decoder network for joint training and recognition.

[0173] Figure 3A An exemplary architecture for implementing the LSD module as an attention-based decoder with an RNN-based decoder according to some embodiments is shown. The attention-based decoder 111 includes a context vector generator 301 and a decoder neural network 303. The context vector generator 301 receives the hidden decoder state 309 from the decoder neural network 303 at the previous time step, the attention weight distribution 307 from the context vector generator 301 at the previous time step, and the alternative representation 121 (i.e., the attention weight distribution 307 described above with reference to FIG. Figure 1AThe context vector generator 301 processes the previous hidden decoder state of the decoder neural network 303, the previous attention weight distribution 307 and the alternative representation 121 to calculate the attention weight distribution over the time frame of the alternative representation 121 and generates a context vector 311 for the current time step as output. The context vector generator 301 provides the context vector 311 for the current time step to the decoder neural network 303.

[0174] For different iterations, the attention-based decoder 111 receives different partitions 315, 317, and 319. For example, the set of partitions includes a first partition 315 and subsequent partitions 317 and 319. The attention-based decoder 111 processes the first partition 315 to generate a first transcription output. After the attention-based neural network completes processing the first partition that places the attention-based network in its internal state, the attention-based decoder 111 processes subsequent partitions 317 and 319 using the attention-based network without resetting the internal state of the attention-based network to successively generate transcription outputs for the subsequent partitions.

[0175] In effect, the attention-based decoder 111 processes different partitions without resetting the internal state of the attention-based network to utilize previously decoded information. Upon determining the end of a speech utterance, the attention-based decoder 111 is configured to reset its internal state.

[0176] In another embodiment, when the attention-based decoder 111 receives the subsequent partitions 317 , 319 , each of the subsequent partitions 317 , 319 resets the internal state of the attention-based decoder 111 to successively produce the transcription output of the subsequent partitions 317 , 319 .

[0177] The decoder neural network 303 receives as input the context vector 311 for the time step and the transcription output 313 and hidden decoder state 309 for the previous time step. The decoder neural network 303 initializes its internal hidden state using the previous hidden decoder state 309 before processing the context vector 311 for the time step and the transcription output 313 from the previous time step to generate as output a set of transcription output scores 313 for the time step. In some implementations, the decoder neural network 303 is a recurrent neural network (RNN) with a softmax output layer. Each transcription output score corresponds to a corresponding transcription output from the set of transcription outputs. For example, as described above with reference to Figure 1AThe transcription output set may be a character or sequence of characters from a Unicode character set used to write one or more natural languages, such as English, Asian, Cyrillic, and Arabic alphabets. The transcription output set may also include Arabic numerals, space characters, and punctuation marks. The score for a given transcription output indicates the likelihood that the corresponding transcription output is the current transcription segment at a time step in the output sequence that is a transcription of the utterance.

[0178] The ASR system processes the transcription output scores 313 for each time step to determine a transcription output sequence representing a transcription of the utterance. For example, for each time step, the speech recognition system may select the transcription output with the highest score from the set of transcription output scores to determine the transcription output sequence.

[0179] Figure 3B An exemplary architecture for implementing the LSD module as an attention-based decoder without hidden decoder states from previous time steps is shown in accordance with some embodiments. Figure 3B The architecture shown is Figure 3A An alternative to the architecture shown in FIG, where the attention-based decoder 111 can be based on a different decoder neural network building block 303 that does not require hidden decoder states from previous time steps. Figure 3B As shown, building block 303 requires a previously transcribed output 325. In some implementations, decoder neural network 323 is based on a self-attention neural network to generate decoder states that are used by encoder-decoder neural network 321 to provide attention to the input encoder state sequence before being fed to post-processing neural network 327. Post-processing neural network 327 generates a transcribed output. This attention-based decoder neural network building block 111 can be repeated D times, where the transcribed output is input to the next decoder building block.

[0180] Example Implementation

[0181] Figure 4A block diagram of a computer-based automatic speech recognition system 400 according to some embodiments is shown. The computer-based speech recognition system 400 includes multiple interfaces that connect the system 400 to other systems and devices. The system 400 includes an input interface 401, which is configured to receive a stream of acoustic frames representing speech utterance features from an input device 403. The input device 403 can be a microphone. Additionally or alternatively, the computer-based speech recognition system 400 can receive acoustic signals from various other types of input interfaces. In some embodiments, the system 400 includes an audio interface that is configured to stream acoustic frames from the acoustic input device 403. In some other embodiments, the input interface includes a network interface controller (NIC) 405 that is configured to stream acoustic frames 409 via a network 407, which can be one or a combination of wired and wireless networks.

[0182] A network interface controller (NIC) 405 is adapted to connect the system 400 to a network 407 via a bus 423, and the network 407 connects the system 400 to a sensing device (e.g., an input device 403 such as a microphone). Additionally or alternatively, the system 400 may include a human-machine interface (HMI) 411. The human-machine interface 411 within the system 400 connects the system 400 to a keyboard 413 and a pointing device 415, wherein the pointing device 415 may include a mouse, a trackball, a touchpad, a joystick, a pointing stick, a stylus, a touch screen, or the like.

[0183] The system 400 includes a processor 421 configured to execute stored instructions 417, and a memory 419 that stores instructions executable by the processor. The processor 421 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 419 can include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 421 can be connected to one or more input and output devices via a bus 423.

[0184] According to some embodiments, instructions 417 may implement a method for end-to-end speech recognition. To this end, computer memory 419 stores encoder 103, frame synchronization decoder (FSD) module 107, synchronization module 109, label synchronization decoder (LSD) module 431, joint scoring module 115, gate 435, and end-of-utterance module 437. In some embodiments, automatic speech recognition system (ASR) 400 is configured for streaming / online recognition. For example, gate 435 is configured to divide a speech utterance into a set of acoustic sequences. For example, in some embodiments, the gate is implemented as a component of input interface 401 that divides speech during speech conversion. The length of each acoustic sequence in the set can be the same or can vary based on characteristics of the speaking speech. In this manner, ASR system 400 transcribes the input acoustic sequences in a streamlined manner. In some embodiments, gate 435 divides the speech utterance into acoustic signal blocks so that the input interface receives one acoustic signal block at a time. For example, gate 435 may be a voice activity detection module that removes non-speech acoustic input.

[0185] The encoder 103 is trained to convert the acoustic frame stream 409 representing the speech utterance features into an encoder state sequence that encodes the acoustic feature frames. In some embodiments, the processor 421 is configured to submit the encoder state sequence to the FSD module 107 to submit the encoder state sequence frame by frame. According to some embodiments, the FSD module 107 can be a connectionist temporal classification (CTC) that is trained to determine the encoder state position in the encoder state sequence that encodes the transcription output. In some embodiments, the FSD module 107 is configured to continuously process each encoder state of the encoded acoustic features to identify the encoder state that updates the prefix list and associated score maintained by the FSD module 107. Each prefix corresponds to a candidate estimate of the decoded transcription output. In addition, in some embodiments, the associated score corresponds to the probability of the transcription output of the prefix. In some other embodiments, the associated score corresponds to the sum of all transcription output probabilities that result in the same prefix.

[0186] The synchronization module 109 is configured to select a portion of the encoder state sequence that includes the encoder state identified by the FSD module 107 in response to an update of the prefix list maintained by the FSD module 107. In some embodiments, the synchronization module 109 is configured to divide the encoder state sequence into a set of partitions based on the positions of the identified encoder states, and submit the set of partitions sequentially to the LSD module 111. In some embodiments, the LSD module 111 can be an attention-based decoder.

[0187] In some embodiments, the LSD module 111 is configured to process a selected portion of the encoder state sequence to update a prefix list maintained by the LSD module 111 using at least one possible estimate of the transcribed output. When the LSD module 111 and the FSD module 107 process the same prefix list, synchronization is achieved between the LSD module decoder 111 and the FSD module 107. Furthermore, decoding the selected portion of the encoder state sequence synchronizes the decoding times of the FSD module 107 and the LSD module 111.

[0188] The joint scoring module 115 is configured to combine the probabilities of corresponding prefixes in the updated prefix list maintained by the FSD module 107 with the probabilities of corresponding prefixes in the updated prefix list maintained by the LSD module 111 to generate a joint prefix list with the best combined probability. In addition, in some embodiments, the joint scoring module 115 is configured to replace the prefix list maintained by the FSD module 107 with the joint prefix list to synchronize the prefixes of the FSD module 107 and the LSD module 111 for subsequent decoding at a subsequent time step. In addition, in some embodiments, the joint scoring module 115 is configured to output the prefix corresponding to the highest joint score as the current portion of the transcription of the incoming stream for the current time step.

[0189] In some embodiments, the memory 419 also stores an end-of-utterance module 437 that is configured to detect the end of a voice utterance. Different embodiments use different techniques to implement the end-of-utterance module 437. For example, some embodiments use a voice activity detection (SAD) module to detect the end of an utterance or a combination of SAD and an auxiliary endpoint detection system.

[0190] The automatic speech recognition system 400 includes an output interface 439 configured to output the transcription output of the system 400. For example, if the transcription output represents characters, the output interface 439 outputs the output character by character. Similarly, if the transcription output represents words, the output interface 439 outputs the output word by word. Additionally or alternatively, in one embodiment, the output interface 439 is configured to accumulate the set of transcription outputs to form words and output each word in the speech utterance individually. Furthermore, in one embodiment, the output interface is configured to output each transcription output individually or together with other outputs.

[0191] In some embodiments, the output interface 439 can display the transcribed output on a display device 441, store the transcribed output in a storage medium, and / or transmit the transcribed output over the network 407. Examples of the display device 441 include a computer monitor, a television, a projector, or a mobile device. The system 400 can also be connected to an application interface 443, which is suitable for connecting the system 400 to external devices 445 for performing various tasks.

[0192] Figure 5 A data flow diagram of an ASR system 100 according to some embodiments is shown. At block 501, the ASR system receives a portion of a speech utterance. The processor is further configured to convert the portion of the speech utterance into a stream of acoustic feature frames representing the speech utterance and submit the stream of acoustic feature frames to an encoder module. The encoder module generates a sequence of encoder states of encoded acoustic feature frames. At block 503, a frame synchronization decoder (FSD) module processes or decodes the sequence of encoder states to identify an encoder state that extends a prefix list using an associated FSD score list maintained by the FSD module. Each prefix in the prefix list is a candidate estimate of a decoded transcription output. Furthermore, each FSD score in the FSD score list corresponds to a probability of the transcription output for the corresponding prefix. In some embodiments, each FSD score in the FSD score list corresponds to the sum of all transcription output probabilities that result in the same prefix. Thus, the FSD module provides information about the position of the encoder state in the sequence of encoder states that encodes a transcription output, such as a character, bit, word, or the like.

[0193] At box 505, the synchronization module uses the position information of the encoder state identified by the FSD module to select a portion of the encoder state sequence that includes the encoder state identified by the FSD module. The synchronization module also submits the current prefix list generated by the FSD module and the selected portion of the encoder state sequence to the label synchronization decoder (LSD) module. This ensures that the LSD module processes the same prefix list as the FSD module in order to estimate the likelihood of the transcription output. This achieves synchronization between the FSD module and the LSD module in the prefix domain. In addition, to ensure that the LSD outputs the LSD score list for the prefix list at approximately the same time as the FSD module, at box 507, only the selected portion of the encoder state sequence is decoded by the LSD module. This enables the LSD module to focus on the selected portion of the encoder state sequence rather than the entire speech utterance, which reduces the output delay of the LSD module. Therefore, synchronization between the FSD module and the LSD module in the time domain is achieved.

[0194] Furthermore, at block 509, the joint scoring module receives prefix lists and associated FSD scores and LSD scores from both the FSD module and the LSD module. The joint scoring module combines the FSD scores for corresponding prefixes in the prefix list maintained by the FSD module with the LSD scores for corresponding prefixes in the prefix list maintained by the LSD module to generate a joint prefix list and an associated joint score list. Furthermore, the prefix probability estimates generated by the external language model 122 can be weighted and added to the corresponding joint scores to further improve the prefix candidate estimates. At block 511, the joint scoring module replaces the prefix list maintained by the FSD module with a joint prefix list that is pruned to remove unlikely prefix candidates and speed up the decoding process. This further synchronizes the lists of possible prefix candidates of the FSD and LSD modules for subsequent decoding of the encoder states at subsequent time steps. Furthermore, at block 513, the joint scoring module outputs the prefix with the highest joint score as the current portion of the transcription of the incoming acoustic frame stream.

[0195] Figure 6 A schematic diagram of an implementation of an FSD module using a prefix beam search module according to some embodiments is shown. In this example, the FSD module includes a CTC neural network 213 that outputs a transcription output probability 610 for each encoder state. The prefix beam search module 620 is configured to search the output 610 of the CTC neural network to produce a list 630 of candidate FSD prefixes with an FSD score above a threshold. In this implementation, each FSD score is the sum of all probabilities of all transcription output sequences that result in the same prefix after collapsing duplicate tags and removing whitespace symbols.

[0196] In this example, the CTC network can predict transcripts directly from the audio input and / or from the encoder state. The CTC network can produce a CTC matrix 610, where the columns correspond to the time steps and each row corresponds to a letter of the alphabet. Since each column sums to 1 and all entries in the CTC matrix are greater than 0, the CTC matrix is a distribution over the alphabet at each time step, essentially a letter prediction. In this example, the alphabet contains at least the letters A-Z, space (_), and blank token (-), where the CTC network requires the latter. Some implementations round small probabilities to zero for readability.

[0197] The simplest decoding approach is to take the letter with the highest probability at each time step—an approach known as maximum decoding or greedy decoding. However, taking the letter with the highest probability at each time step is a simplification because, given a contraction rule, it is possible to trace multiple paths through the CTC matrix that all produce the same label.

[0198] To this end, some embodiments use the prefix beam search module 620 to consider not only the highest probability, but also the sum of all probabilities of all transcribed output sequences that result in the same prefix after collapsing duplicate tags and removing whitespace symbols. In this way, the prefix beam search module calculates the FSD score of the candidate FSD prefix list based on the FSD score of the current FSD prefix list pruned according to the joint score. In some implementations, the prefix beam search module 620 also uses the scores generated by one or more external language models 122.

[0199] Figure 7 A block diagram of some components that can be used in various configurations for implementing a system according to some embodiments is shown. For example, component 700 can include a hardware processor 701 in communication with a sensor 703 or a sensor such as an acoustic sensor that collects data including acoustic signals 709 from an environment 711. In addition, sensor 703 can convert acoustic input into acoustic signals 709. Hardware processor 701 is in communication with computer storage memory (i.e., memory 713), such that memory 713 includes stored data including algorithms, instructions, and other data that can be implemented by hardware processor 701.

[0200] Optionally, the hardware processor 701 can be connected to a network 715 that communicates with a data source 717, a computer device 719, a mobile phone device 721, and a storage device 723. Also optionally, the hardware processor 701 can be connected to a network-enabled server 725 that is connected to a client device 727. The hardware processor 701 can optionally be connected to an external memory device 729 and / or a transmitter 731. In addition, the speaker's text can be output according to a particular user intended use 733. For example, some types of user uses may include displaying the text on one or more display devices such as a monitor or screen, or inputting the speaker's text into a computer-related device for further analysis, etc.

[0201] It is contemplated that hardware processor 701 may include one or more hardware processors depending on the requirements of a particular application, where the processor may be internal or external. Of course, other components may be combined with component 700, including output interfaces and transceivers, among other devices.

[0202] By way of non-limiting example, network 735 can comprise one or more local area networks (LANs) and / or wide area networks (WANs), wherein the network environment can be similar to an enterprise-wide computer network, an intranet, and the Internet. Considering all components mentioned, there can be any number of client devices, storage components, and data sources used together with component 700. Each can comprise a single device or multiple devices collaborating in a distributed environment. In addition, component 700 can comprise one or more data sources 717. Data source 717 comprises a data resource for training a speech recognition network. The data provided by data source 717 can comprise labeled data and unlabeled data, such as transcribed and untranscribed data. For example, in one embodiment, the data comprises one or more sounds and can also comprise corresponding transcription information or labels that can be used to initialize the speech recognition network.

[0203] In addition, the unlabeled data in the data source 717 can be provided by one or more feedback loops. For example, usage data from a spoken search query performed on a search engine can be provided as untranscribed data. Other examples of data sources can include, but are not limited to, various spoken audio or image sources, including streaming sound or video, web queries, mobile device camera or audio information, webcam feeds, smart glasses and smartwatch feeds, customer care systems, security camera feeds, web documents, directories, user feeds, SMS logs, instant message logs, spoken text transcripts, gaming system user interactions such as voice commands or captured images (e.g., depth camera images), tweets, chat or video call logs, or social network media. The specific data source 717 used can be determined based on the application, including whether the data is a specific category of data (e.g., data related only to a specific type of sound, such as a machine system, entertainment system) or is general (non-specific category) in nature.

[0204] Component 700 may include or be connected to a third-party device, which may include any type of computing device that may be interested in having an automatic speech recognition (ASR) system on the computing device. For example, the third-party device may include a computer device 719 or a mobile device 721. It is expected that the user device can be implemented as a personal digital assistant (PDA), a mobile device such as a smart phone, a smart watch, smart glasses (or other wearable smart device), an augmented reality headset, a virtual reality headset. In addition, the user device can be a laptop computer such as a tablet computer, a remote control, an entertainment system, a vehicle computer system, an embedded system controller, an appliance, a home computer system, a security system, a consumer electronic device, or other similar electronic device. In one embodiment, the client device 727 is capable of receiving input data, such as audio and image information that can be used by the ASR system operating on the device described herein. For example, the third-party device may have a microphone or line-in for receiving audio information, a camera for receiving video or image information, or a communication component (e.g., Wi-Fi capability) for receiving such information from another source such as the Internet or data source 717.

[0205] An ASR model using a speech recognition network can process the input data to determine computer-usable information. For example, a query spoken by a user into a microphone can be processed to determine the content of the query, such as if a question was asked. Exemplary third-party devices 719 and 721 are optionally included in component 700 to illustrate an environment in which a deep neural network model can be deployed. In addition, some embodiments of the present disclosure may not include third-party devices 737 and 739. For example, the deep neural network model can be on a server or in a cloud network, system, or similar arrangement.

[0206] Regarding storage 723, storage 723 can store information including data, computer instructions (e.g., software program instructions, routines, or services), and / or models used in embodiments of the technology described herein. For example, storage 723 can store data from one or more data sources 717, one or more deep neural network models, information for generating and training deep neural network models, and computer-usable information output by one or more deep neural network models.

[0207] Implementation Method

[0208] The description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. It is contemplated that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0209] Specific details are provided in the following description to provide a comprehensive understanding of the embodiments. However, those skilled in the art will appreciate that the embodiments may be practiced without these specific details. For example, the systems, processes, and other elements of the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other examples, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. In addition, the same reference numerals and designations in the various figures indicate the same elements.

[0210] Furthermore, various embodiments may be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not discussed or included in the diagram. Furthermore, not all operations in any particularly described process will occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to the function returning to the calling function or the main function.

[0211] Furthermore, embodiments of the disclosed subject matter can be implemented at least in part manually or automatically. Manual or automatic implementation can be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks can be stored in a machine-readable medium. A processor can perform the necessary tasks.

[0212] In addition, the embodiments of the present disclosure and the functional operations described in this specification can be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. In addition, some embodiments of the present disclosure can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by a data processing device or for controlling the operation of the data processing device. In addition, the program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, optical signal, or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more thereof.

[0213] According to an embodiment of the present disclosure, the term "data processing equipment" may include all kinds of devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may include special-purpose logic circuits, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0214] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. The program can be stored in a portion of a file that preserves other programs or data, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordination files (for example, files storing one or more modules, subroutines or code portions). The computer program can be deployed to execute on a computer or on multiple computers located at a site or distributed across multiple sites and interconnected by a communication network. The computer suitable for executing the computer program can, for example, be based on a general or special purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or carrying out instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include or be operably connected to receive data from one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or to transfer data to one or more mass storage devices or both. However, a computer need not have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, etc.

[0215] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, as well as a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0216] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server; or includes a middleware component, such as an application server; or includes a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0217] A computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0218] Although the present disclosure has been described with reference to certain preferred embodiments, it will be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure. It is therefore intended that the appended claims cover all such changes and modifications as fall within the true spirit and scope of the present disclosure.

Claims

1. A computer-based automatic speech recognition system, wherein: The system utilizes a processor coupled with stored instructions implementing modules of the system, wherein the processor executing the instructions is configured to: encoding an incoming stream of acoustic frames representing features of a speech utterance into an encoder state sequence of encoded acoustic feature frames; and Continuously processing each encoder state of the encoded acoustic feature frame with a frame synchronization decoder (FSD) module until a termination condition is satisfied, wherein the FSD module, in response to identifying an encoder state that carries information about a new transcription output, expands a current FSD prefix list to generate a candidate FSD prefix list and a corresponding FSD score, wherein each FSD prefix in the candidate list is an estimate by the FSD module of a decoded transcription output in an encoder state processed by the FSD module, wherein a probability of the FSD prefix being the decoded transcription output is defined by the corresponding FSD score, wherein, in response to generating the candidate FSD prefix list, the processor is configured to: selecting a portion of the encoder state that includes the encoder state identified by the FSD module and triggering a tag synchronized decoder (LSD) module to process the selected portion of the encoder state to determine an LSD score according to the LSD module, the LSD score defining a probability that an FSD prefix in the candidate FSD prefix list is the decoded transcription output in the selected portion of the encoder state; pruning the candidate FSD prefix list according to a joint score defined by a combination of the corresponding FSD score and the LSD score; and The current FSD prefix list is replaced with the pruned FSD prefix list to continue decoding the next transcription output.

2. The system according to claim 1, wherein: The size of the current FSD prefix list is fixed and smaller than the size of the candidate FSD prefix list, such that pruning retains n best FSD prefixes with the highest joint scores in the candidate FSD prefix list, where n is the size of the current FSD prefix list.

3. The system according to claim 1, wherein: The processor replaces only the FSD prefix without replacing the FSD score in the replaced current FSD prefix list.

4. The system according to claim 1, wherein: The processor is configured to output the FSD prefix having the highest joint score as a decoded transcription output of a portion of the incoming acoustic frame stream corresponding to the portion of the encoder state sequence.

5. The system according to claim 1, wherein: The FSD module includes: A connectionist temporal classification (CTC) neural network that outputs the probability of the transcription output for each encoder state; and a prefix beam search module configured to search the output of the CTC neural network to produce a candidate FSD prefix list having an FSD score above a threshold, wherein each FSD score is the sum of all probabilities of all transcribed output sequences that result in the same prefix after collapsing duplicate tags and removing whitespace symbols, wherein the prefix beam search module calculates the FSD score of the candidate FSD prefix list based on the FSD score of the current FSD prefix list pruned according to the joint score.

6. The system according to claim 1, wherein: The FSD module includes a neural network with a recurrent neural network transducer (RNN-T) architecture.

7. The system according to claim 1, wherein: The LSD module includes an attention-based decoder neural network.

8. The system according to claim 7, wherein: The LSD module determines a corresponding LSD score and a list of LSD prefixes for each partition of the encoder state submitted as input, and outputs LSD scores only for LSD prefixes that match the FSD prefix.

9. The system according to claim 1, wherein: The modules of the system include a synchronization module configured to synchronize the FSD module and the LSD module in the time domain and the prefix domain by triggering the LSD module to process a portion of the encoder state including a new transcription output identified by the FSD module and by causing the LSD module to estimate an LSD score for an FSD prefix generated by the FSD module.

10. The system according to claim 9, wherein: The synchronization module selects the portion of the sequence of encoder states from the beginning of the sequence up to a leading encoder state, wherein the leading encoder state is determined by shifting the position of the encoder state identified by the FSD module forward by a fixed shift.

11. The system according to claim 9, wherein: The synchronization module selects a portion of the encoder state sequence that includes a predetermined number of look-ahead encoder states and a predetermined number of look-back encoder states relative to a position of an encoder state identified by the FSD module.

12. The system according to claim 1, wherein: The modules of the system include an encoder neural network trained jointly using the FSD module and the LSD module to form a triggered attention TA neural network.

13. The system according to claim 12, wherein: The encoder includes a unidirectional encoder neural network based on a parallel delayed long short-term memory (PTDLSTM) stream.

14. The system according to claim 12, wherein: The encoder and the LSD module are implemented using a transformer architecture.

15. A non-transitory computer-readable storage medium having embodied thereon a program comprising a module executable by a processor for performing automatic speech recognition, the module comprising: an encoder configured to encode an incoming stream of acoustic frames representing speech utterance features into an encoder state sequence of encoded acoustic feature frames; a frame synchronization decoder FSD module configured to continuously process each encoder state of the encoded acoustic feature frame to expand a current FSD prefix list in response to identifying an encoder state that carries information about a new transcription output to produce a candidate FSD prefix list and a corresponding FSD score, each FSD prefix in the candidate list being an estimate by the FSD module of a decoded transcription output in the encoder state processed by the FSD module, wherein a probability that the FSD prefix is the decoded transcription output is defined by the corresponding FSD score; a synchronization module configured to select a portion of the encoder state including the encoder state identified by the FSD module; a tag-sync decoder (LSD) module configured to process a selected portion of an encoder state to combine, according to the LSD module, an FSD score for the FSD prefix with a corresponding LSD score defining a probability that the FSD prefix is output by the decoded transcription; a joint scoring module configured to prune the FSD prefixes according to the combined FSD score and LSD score, and replace the current FSD prefix list with the pruned FSD prefix list to continue decoding the next transcription output; and An output interface configured to output the FSD prefix having the highest joint score as the decoded transcription output of the incoming acoustic frame stream of the encoder state processed by the FSD module.

16. The medium according to claim 15, wherein The synchronization module is configured to synchronize the FSD module and the LSD module in the time domain and the prefix domain by triggering the LSD module to process the portion of the encoder state including the new transcription output identified by the FSD module and by passing the FSD prefix to the LSD module so that the LSD module can estimate the LSD score of the FSD prefix.

17. The medium according to claim 15, wherein The FSD module includes: A connectionist temporal classification (CTC) neural network that outputs the probability of the transcription output for each encoder state; and a prefix beam search module configured to search the output of the CTC neural network to generate a candidate FSD prefix list having an FSD score above a threshold, wherein each FSD score is the sum of all probabilities of all transcribed output sequences that result in the same prefix after collapsing duplicate tags and removing whitespace symbols, wherein the prefix beam search module calculates the FSD score of the candidate FSD prefix list based on the FSD score of the current FSD prefix list pruned according to the joint score; and Among them, the LSD module includes an attention-based decoder neural network.

18. The medium according to claim 15, wherein The module further comprises: an external language model configured to score the candidate FSD prefix list, wherein the joint scoring module updates a joint FSD score and an LSD score with the score determined by the external language model, and prunes the candidate FSD prefix list based on the updated joint FSD and LSD scores.

19. A method for automatic speech recognition, wherein: The method uses a processor coupled to stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising the steps of: encoding an incoming stream of acoustic frames representing speech utterance features into an encoder state sequence of encoded acoustic feature frames; continuously processing each encoder state of the encoded acoustic feature frame with a frame synchronization decoder (FSD) module until a termination condition is satisfied, wherein the FSD module, in response to identifying an encoder state that carries information about a new transcription output, expands a current FSD prefix list to produce a candidate FSD prefix list and a corresponding FSD score, wherein each FSD prefix in the candidate list is an estimate by the FSD module of a decoded transcription output in the encoder state processed by the FSD module, and wherein a probability of the FSD prefix being the decoded transcription output is defined by the corresponding FSD score; selecting a portion of the encoder state that includes the encoder state identified by the FSD module; triggering a tag synchronized decoder (LSD) module to process the selected portion of the encoder state to determine an LSD score according to the LSD module, the LSD score defining a probability that an FSD prefix in the candidate FSD prefix list is the decoded transcription output in the selected portion of the encoder state; pruning the candidate FSD prefix list according to a joint score defined by a combination of the corresponding FSD score and the LSD score; and Replace the current FSD prefix list with the pruned FSD prefix list to continue decoding the next transcription output.

20. The method according to claim 19, further comprising the steps of: The FSD prefix having the highest joint score is output as a decoded transcription output of a portion of the incoming acoustic frame stream corresponding to the portion of the encoder state sequence.

Images