End-to-end speech recognition adapted to multi-speaker applications

Abstract

A system for performing end-to-end automatic speech recognition (ASR). The system is configured to collect a sequence of acoustic frames associated with a mixture of utterances performed by multiple speakers. Each frame from the sequence of acoustic frames is encoded using a multi-head encoder that encodes each frame into a transcription output likelihood and a speaker identity likelihood. The multi-head encoder thus generates a sequence of transcription output likelihoods and a sequence of speaker identity likelihoods corresponding to the sequence of acoustic frames, which are decoded using a decoder that performs an alignment operation to generate a sequence of transcription outputs annotated with speaker identities to perform speaker separation.

Description

[Technical Field]

[0001] The present disclosure relates generally to artificial intelligence (AI) systems for speech recognition, and more particularly to methods and systems for end-to-end speech recognition adapted for multi-speaker applications. [Background technology]

[0002] Neural networks can reproduce and model nonlinear processes, which has led to their use in numerous applications in various fields over the past few decades. A neural network can learn (or train) by processing examples, each containing a known "input" and "outcome," forming probability-weighted associations between the two, which are stored within the network's own data structure. Training a neural network from a given example is typically done by determining the difference between the network's processed output (often a prediction) and a target output, also referred to herein as a training label. This difference represents the error that the training seeks to reduce. The network then uses this error value to adjust its weighted associations accordingly, according to a learning rule. Successive adjustments cause the neural network to generate outputs that increasingly resemble the target output. After a sufficient number of these adjustments, training can be terminated based on certain criteria.

[0003] This type of training is typically called supervised learning. During supervised learning, neural networks "learn" to perform a task by considering examples and are generally not programmed with task-specific rules. For example, in image recognition, a neural network may learn to identify images containing cats by analyzing example images labeled "cat" or "no cat" and using the results to identify cats in other images. The neural network does this without any prior knowledge of cats, such as that cats have fur, tails, whiskers, and cat-like faces. Instead, neural networks automatically generate discriminative characteristics from the examples they process.

[0004] However, to perform such supervised learning, images must be labeled as cats or dogs. Such labeling is a tedious and laborious process. Also, in this image recognition example, the labeling is unambiguous: an image contains either a cat, a dog, or neither a cat nor a dog. Such unambiguous labeling is not always possible. For example, some training applications tackle sequence problems with variable timing. The time variable can create one-to-many or many-to-one ambiguities in such training, where the input sequence has a different length than the output sequence.

[0005] Specifically, some methods for training neural networks use the connectionist time series classification (CTC) objective function algorithm. CTC is a loss function used to train neural networks when there is no temporal alignment information available between a sequence of training labels and a longer sequence of label probabilities output by the neural network, which is computed from a sequence of observations input to the neural network. Such missing temporal alignment information creates a temporal ambiguity between the sequence of label probabilities output by the neural network and the supervised information for training, which is a sequence of training labels that can be resolved using the CTC objective function.

[0006] However, the CTC objective function is only suitable for resolving temporal ambiguity during neural network training. If other types of ambiguity need to be considered, the CTC objective function will not be useful.

[0007] A generalized form of the CTC objective function is graph-based time series classification (GTC), a type of loss function that uses graph representations in the loss function for training deep neural networks. The GTC loss function is used to handle sequence-to-sequence temporal alignment disambiguation using deep neural networks. GTC can take graph-based supervised information as input to describe all possible alignments between input and output sequences to learn the best possible alignment from the training data.

[0008] One example of a sequence-based input to a neural network that requires temporal and label disambiguation is audio input, which can be in the form of utterances from one or more speakers that may need to be identified and separated for speech applications.

[0009] One example of such a voice application is automatic speech recognition (ASR) systems, which are widely deployed for various interface applications such as speaker-based voice search. However, creating a speech recognition system with high recognition accuracy is difficult because it requires deep linguistic knowledge of the target language the ASR system accepts. For example, a set of phonemes, a vocabulary, and a pronunciation lexicon are essential for creating such an ASR system. The phoneme set must be carefully defined by a linguist of the language. The pronunciation lexicon must be created manually by assigning one or more phoneme sequences to each word in a vocabulary containing over 100,000 words. Furthermore, some languages ​​do not have explicit word boundaries and therefore may require tokenization to create a vocabulary from a text corpus. Therefore, developing a speech recognition system, especially for minority languages, is extremely difficult. Another problem is that speech recognition systems are factored into several modules, including acoustics, lexicons, and language models, which are optimized separately. This architecture, in which each model is trained to match other models, can lead to local optimization.

[0010] End-to-end and sequence-to-sequence neural network models have each gained interest and popularity in the ASR field in recent years. The output of an end-to-end ASR system is typically a grapheme sequence, which can be either a single character or larger units such as word fragments and whole words. The appeal of end-to-end ASR is that it allows for a simplified system architecture compared to traditional ASR systems by being composed of neural network components and by avoiding the need for linguistic expertise to build the ASR system.

[0011] End-to-end ASR systems can directly train all components of a speech recognizer, including pronunciation, acoustic, and language models, which avoids the need for language-specific linguistic information and text normalization. These ASR systems perform sequence-to-sequence conversion, where the input is a sequence of acoustic features extracted from speech frames at a certain rate, and the output is a sequence of characters. Sequence-to-sequence conversion allows for various linguistic characteristics to be taken into account to improve the quality of recognition.

[0012] However, the improvement in the quality of end-to-end ASR systems comes at the cost of output delay caused by the need to accumulate sequences of acoustic features and / or sequences of acoustic frames for simultaneous recognition. Therefore, end-to-end ASR systems are less applicable to online / streaming ASR where low latency is required.

[0013] Various techniques, such as triggered attention or limited self-attention, have been developed to reduce output delay in end-to-end ASR systems. See, for example, U.S. Patent No. 11,100,920. However, these techniques are not applicable, or at least not directly applicable, to multi-speaker recognition and / or multi-speaker streaming applications. This is because multi-speaker applications involve two independent tasks: speaker separation and speech recognition. Currently, speaker separation in multi-speaker ASR systems is either a pre-processing or post-processing technique that introduces additional delays that current methods for end-to-end streaming speech recognition cannot handle.

[0014] Therefore, there is a need to reduce the output delay in multi-speaker applications suitable for end-to-end and / or sequence-to-sequence speech recognition applications. Summary of the Invention

[0015] An objective of some embodiments is to reduce output delay in multi-speaker applications configured for end-to-end and / or sequence-to-sequence speech recognition applications. An example of such an application is a streaming speech recognition application. Some embodiments are based on the understanding that to reduce delay in multi-speaker speech recognition applications, the speech separation task and the speech recognition task should be considered together, such that speech recognition is performed simultaneously with speech separation. Doing so can prevent additional delays in speech recognition caused by speech separation pre-processing or post-processing techniques.

[0016] Additionally or alternatively, some embodiments are based on the recognition that speech separation can be replaced with speaker identification when considered in conjunction with speech recognition. In contrast to speech separation, which is considered an independent task, speaker identification can be treated as a task subordinate to speech recognition. Thus, speaker identification can be implemented as an internal process of speech recognition.

[0017] With that in mind, some embodiments are based on the understanding that speech recognition in an end-to-end ASR system is typically performed using an encoder and decoder. To make speaker identification an internal process, some embodiments enable the decoder to decode both the speech and the speaker, while performing speaker identification at the encoder level. In this way, speech separation is transformed into a part of the decoding that does not incur additional delay. However, to achieve this effect, the encoder needs to be a multi-head or multi-output encoder that generates symbol encodings and speaker identities for each speech frame.

[0018] Some embodiments recognize that having supervision information from a directed graph of nodes connected by edges representing labels and transitions between labels allows for the imposition of flexible rules for training a neural network. For example, some embodiments disclose training a neural network with a GTC objective function without inserting a blank label between all training labels or by using multiple different blank labels. Additionally or alternatively, some embodiments disclose training a neural network with a GTC objective function that uses a Hidden Markov Model (HMM)-like topology for each label, which may include multiple states. Additionally or alternatively, some embodiments disclose training a neural network with a GTC objective function that uses a directed graph with transitions between nodes associated with costs or weighting factors.

[0019] In addition to using supervision information on the directed graph, some embodiments modify the GTC objective function to accommodate label alignment. For example, the GTC objective function is defined by maximizing the sum of the conditional probabilities of all node sequence paths with specific start and end nodes that can be generated from a given directed graph by expanding the directed graph to the length of the sequence of label probabilities output by the neural network. GTC training loss and gradients can be efficiently computed by a dynamic programming algorithm based on the calculation of forward and backward variables and the joint stitching of both.

[0020] GTC-based training of a neural network aims to update trainable parameters of the neural network by optimizing the neural network's label predictions so that the best overall predicted sequence of labels can be generated for a set of training sample and graph-based labeling information pairs using a directed graph that minimizes the error of all possible label sequence predictions. Trainable parameters include, for example, the neural network's neuron weights, hyperparameters, etc.

[0021] Additionally or alternatively, some embodiments recognize that the GTC objective function and directed graph allow for consideration of not only sequences of multiple labels, but also different probabilities for sequences of multiple labels. Such consideration is advantageous for the GTC objective function because it allows for tailoring the supervision information to specific situations. To that end, in some embodiments, the directed graph is weighted with different weights for at least some edges or transitions. These transition weights are used to calculate the conditional probabilities of sequences of labels.

[0022] Some embodiments are based on the recognition that GTC can be used to encode an N-best list of pseudo-label sequences into a graph for semi-supervised learning. To that end, some embodiments disclose an extension of GTC to model the posterior probabilities of both labels and label transitions using neural networks, which can be applied to a wider range of tasks. The extended GTC (GTC-e) is used for multi-speaker speech recognition tasks. Transcriptions and speaker information of multi-speaker utterances are represented by a graph, with speaker information associated with transitions and ASR outputs associated with nodes. Using GTC-e, multi-speaker ASR modeling becomes very similar to single-speaker ASR modeling in that tokens from multiple speakers are recognized as a single merged sequence in chronological order.

[0023] Furthermore, methods for training neural network models using loss functions to learn to map input sequences to typically shorter output sequences, such as CTC and recurrent neural network transducers (RNN-T), are commonly used in automatic speech recognition (ASR) technology. CTC and RNN-T losses are designed for alignment-free training of neural network models, learning to map input sequences (e.g., acoustic features) to typically shorter sequences of output labels (e.g., word or subword units). While CTC loss requires neural network outputs to be conditionally independent, RNN-T loss provides an extension for training neural networks whose output frames conditionally depend on previous output labels. To perform training without knowing the alignment between the input and output sequences, both loss types marginalize over the set of all possible alignments. Such alignments are induced from supervision information (sequences of labels) by applying specific instructions that define how to expand the sequence of labels to adjust to the length of the input sequence. In both cases, such instructions include the use of additional blank labels and loss-type specific transition rules.

[0024] However, modifying the training lattice of the transducer model to achieve a strictly monotonic alignment between input and output sequences may leave other aspects of the RNN-T unchanged, such as the emission of ASR labels over a single time frame.

[0025] Some embodiments are based on the realization of the GTC-Transducer (GTC-T) objective function, which extends GTC to a conditionally dependent neural network with output similar to RNN-T. In one embodiment, GTC-T allows users to define label transitions in a graph format, thereby easily exploring new lattice structures for transducer-based ASR. In one embodiment, a CTC-like lattice is used to train a GTC-T-based ASR system. Furthermore, the GTC-T objective function allows the use of different graph topologies for constructing the training lattice, for example, a graph type corresponding to a CTC-like topology or a graph type corresponding to a MonoRNN-T (or RNA) loss type.

[0026] Accordingly, one embodiment discloses an end-to-end automatic speech recognition (ASR) system including a processor and a memory having instructions stored thereon. The processor is configured to execute the stored instructions to cause the ASR system to collect a sequence of acoustic frames providing a digital representation of an acoustic signal including a mixture of utterances performed by multiple speakers. The processor is further configured to encode each frame from the sequence of acoustic frames using a multi-head encoder that encodes each frame into a transcription output likelihood and a speaker identity likelihood to generate a sequence of transcription output likelihoods and a sequence of speaker identity likelihoods corresponding to the sequence of acoustic frames. The processor is further configured to decode the sequence of transcription output likelihoods and the sequence of speaker identity likelihoods with a decoder that performs alignment to generate a sequence of transcription outputs annotated with the speaker identities. Additionally, the processor is configured to submit the sequence of transcription outputs annotated with the speaker identities to a downstream application.

[0027] Accordingly, one embodiment discloses a computer-implemented method for performing end-to-end ASR. The method includes collecting a sequence of acoustic frames providing a digital representation of an acoustic signal containing a mixture of utterances performed by multiple speakers. The method further includes encoding each frame from the sequence of acoustic frames using a multi-head encoder that encodes each frame into a transcription output likelihood and a speaker identity likelihood to generate a sequence of transcription output likelihoods and a sequence of speaker identity likelihoods corresponding to the sequence of acoustic frames. The method further includes decoding the sequence of transcription output likelihoods and the sequence of speaker identity likelihoods with a decoder that performs alignment to generate a sequence of transcription outputs annotated with the speaker identities. Additionally, the method includes submitting the sequence of transcription outputs annotated with the speaker identities to a downstream application. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a block diagram illustrating an end-to-end automatic speech recognition (ASR) system in accordance with an exemplary embodiment. [Figure 2] 2 illustrates a block diagram of the internal components of the end-to-end ASR system of FIG. 1, in accordance with an exemplary embodiment. [Figure 3] FIG. 1 illustrates an example architecture of an end-to-end ASR system according to an example embodiment. [Figure 4] 1 illustrates an extension of GTC for an end-to-end ASR system to perform multiple speaker separation, according to an exemplary embodiment. [Figure 5] FIG. 2 illustrates the architecture of the end-to-end ASR system of FIG. 1 using a neural network trained with the GTC-e objective function, according to an exemplary embodiment. [Figure 6]6 illustrates an example operation of the neural network of FIG. 5, according to an exemplary embodiment. [Figure 7A] FIG. 1 is a schematic diagram illustrating a workflow for training a neural network using a graph-based time series classification (GTC) objective function, according to an example embodiment. [Figure 7B] 10 illustrates a sequence of probability distributions output by a neural network in accordance with an illustrative embodiment; [Figure 7C] 1 illustrates an exemplary directed graph, in accordance with an exemplary embodiment; [Figure 7D] 10 illustrates an example of possible unconstrained repetition of labels during the evolution of a directed graph, according to an illustrative embodiment. [Figure 7E] 1 illustrates an exemplary monotonic directed graph, in accordance with an illustrative embodiment. [Figure 7F] 10 illustrates a monotone directed graph modified based on constraints on label repetition, in accordance with an example embodiment. [Figure 8] FIG. 1 illustrates method steps for training a neural network using a GTC objective function, according to an example embodiment. [Figure 9] 1 illustrates a beam search algorithm used during a neural network decoding operation, according to an example embodiment. [Figure 9 Continued] 1 illustrates a beam search algorithm used during a neural network decoding operation, according to an example embodiment. [Figure 10] Table 1 shows the greedy retrieval results for word error rate (WER) using the GTC-e objective function compared to other methods, according to an exemplary embodiment. [Figure 11] Table 2 shows the greedy search results for the ASR performance of the ASR system based on the GTC-e objective function based on the oracle token error rate, according to an exemplary embodiment. [Figure 12] Table 3 shows beam search results for ASR performance of the ASR system based on the GTC-e objective function based on WER, according to an exemplary embodiment. [Figure 13]Table 4 shows beam search results for ASR performance of an ASR system based on the GTC-e objective function based on WER for multiple speakers, according to an exemplary embodiment. [Figure 14A] 1 illustrates a neural network architecture for an ASR system implemented using a GTC-T objective function, according to an example embodiment. [Figure 14B] 10 illustrates pseudocode for a beam search algorithm for GTC-T with CTC-like graphs, in accordance with an example embodiment. [Figure 14C] 10 illustrates a comparison of ASR results for CTC, RNN-T, and GTC-T losses on the HKUST benchmark, according to an example embodiment. [Figure 14D] 10 illustrates a comparison of ASR results for CTC, RNN-T, and GTC-T losses on the LibriSpeech dataset benchmark, according to an example embodiment. [Figure 15] FIG. 1 illustrates a block diagram of a computer-based system trained using a GTC-e objective function, in accordance with an illustrative embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0029] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0030] As used in this specification and claims, the words "for example," "for example," "e.g.," "etc.", and the verbs "comprise," "have," "include," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended, meaning that the list should not be viewed as excluding other additional components or items. The phrase "based on" means based at least in part on. Furthermore, it should be understood that the phraseology and terminology used herein are for purposes of description and should not be considered limiting. Any headings used within this description are for convenience only and have no legal or limiting effect.

[0031] In recent years, dramatic progress has been achieved in automatic speech recognition (ASR), particularly through the exploration of neural network architectures that improve the robustness and generalization ability of ASR models. The emergence of end-to-end ASR models has simplified ASR architectures with frameworks such as connectionist time series classification (CTC), attention-based encoder-decoder models, and recurrent neural network-transducer (RNN-T) with a single neural network. In addition, graph modeling has traditionally been used in ASR, such as using hidden Markov model (HMM)-based systems, which combine several modules including pronunciation lexicons, context-dependent relationships, and language models using weighted finite-state transducers (WFSTs). Recently, the use of graph representations in loss functions for training deep neural networks has also been proposed, with a new loss function called graph-based time series classification (GTC), which is a generalization of CTC to handle sequence-to-sequence problems. GTC takes graph-based supervision as input and can describe all possible alignments between input and output sequences to learn the best possible alignment from training data.

[0032] GTC is used to enhance ASR performance through semi-supervised training by using an N-best list of ASR hypotheses that are converted into a graph representation to train an ASR model using unlabeled data. However, in the original GTC, only the posterior probabilities of ASR labels are trained, and trainable label transitions are not considered.

[0033] Some embodiments are based on the recognition that extending GTC to handle label transitions would allow modeling of information about labels. For example, in the case of multi-speaker speech recognition, where some overlap between the speech signals of multiple speakers is considered, transition weights may be used to model speaker predictions that are aligned with ASR label predictions at the frame level, so that once an ASR label is predicted, it can also be detected whether it belongs to a particular speaker.

[0034] 1 is a block diagram 100 illustrating an end-to-end ASR system 104 according to an example embodiment. The end-to-end ASR system 104 includes a memory 105 having instructions stored therein. The instructions are executed by a processor 106 to cause the end-to-end ASR system 104 to perform certain operations. The operation of the end-to-end ASR system 104 is described below in the form of various embodiments.

[0035] In one embodiment, the end-to-end ASR system 104 is configured to collect a sequence of acoustic frames that provide a digital representation of an acoustic signal including a mixture of utterances performed by multiple speakers. For example, a first speaker 101 outputs a first audio signal, and a second speaker 102 outputs a second audio signal, which are superimposed, and a superimposed utterance 103 corresponding to the mixture of the utterances of the first speaker 101 and the second speaker 102 is collected by the end-to-end ASR system 104. The end-to-end ASR system 104 includes an input interface that converts the superimposed utterance into a digital representation of an acoustic signal corresponding to the sequence of frames in the superimposed utterance 103.

[0036] Thus, the overlapped utterances 103 correspond to input acoustic sequences that are processed by the end-to-end ASR system 104 to generate a sequence of transcription outputs 107 annotated with speaker identities that are submitted to a downstream application. Each transcription output sequence is a transcription of an utterance or a portion of the utterance represented by the corresponding input acoustic signal. For example, the end-to-end ASR system 104 can take the overlapped utterances 103 (hereinafter also referred to as acoustic signals) and generate a corresponding transcription output 107 that is a transcription of the utterance represented by the input acoustic signal 103 and annotated with speaker IDs for at least one of multiple speakers, such as a first speaker 101 or a second speaker 102.

[0037] The input acoustic signal 103 may include a sequence of frames of audio data, e.g., a continuous stream of data, that are a digital representation of speech. The sequence of frames of audio data may correspond to a sequence of time steps, e.g., each frame of audio data is associated with 25 milliseconds of audio stream data that is shifted in time by an additional 10 milliseconds from the previous frame of audio data. Each frame of audio data in the sequence of frames of audio data may include feature values ​​that characterize the portion of the utterance at the corresponding time step for that frame. For example, the sequence of frames of audio data may include filterbank spectral feature vectors.

[0038] The end-to-end ASR system 104 receives an input acoustic sequence and processes the input acoustic sequence to generate transcription output sequences. Each transcription output sequence is a transcription of an utterance or a portion of the utterance represented by a corresponding input acoustic signal. For example, the end-to-end ASR system 104 may receive an input acoustic signal 103 corresponding to a mixture of acoustic signals from multiple speakers, such as a first speaker 101 and a second speaker 102, and generate corresponding transcription outputs 107 in time sequence that are transcriptions of the utterance represented by the input acoustic signals 103 from the multiple speakers.

[0039] The transcription output 107 may include a sequence of transcription fragments of the speech represented by the input acoustic signal 103. The transcription output may include one or more characters. For example, the transcription output may be a character or sequence of characters from the Unicode character set. For example, the character set may include English, Asian, Cyrillic, and Arabic alphabets. The character set may also include Arabic numerals, white space, and punctuation marks. Additionally or alternatively, the transcription output may include bits, words, and other linguistic structures.

[0040] To that end, the end-to-end ASR system 104 is configured to perform a series of operations including encoding, decoding, and outputting operations, the examples of which are shown in FIG.

[0041] 2 shows a block diagram 200 of the internal components of the end-to-end ASR system 104 according to some embodiments of the present disclosure. The end-to-end ASR system 104 includes an encoder 201, a decoder 202, and an output generation module 203. The encoder 201, the decoder 202, and the output generation module 203 are examples of operations performed by the end-to-end ASR system 104, which are performed by the processor 106 executing their respective stored computer instructions.

[0042] The encoder 201 is a multi-head encoder, with one head corresponding to each speaker of multiple speakers, such as the first speaker 101 and the second speaker 102. The encoder 201 is configured to cause the end-to-end ASR system 104 to encode each frame in an input sequence of acoustic frames of the input acoustic signal 103 with the multi-head encoder 201, which encodes each frame into a transcription output likelihood and a speaker identity likelihood to generate a sequence of transcription output likelihoods and a sequence of speaker identity likelihoods corresponding to the sequence of acoustic frames of the input acoustic signal 103.

[0043] Further, decoder 202 is configured to decode the sequence of likelihoods of transcription outputs and the sequence of likelihoods of speaker identities provided by encoder 201. Decoder 202 is an alignment-based decoder for generating an alignment of the sequence of transcription outputs annotated with speaker identities.

[0044] The sequence of transcription outputs annotated with speaker identities is submitted by the output generation module 203 to a downstream application as transcription output 107. The downstream application may be an online streaming-based application such as an online music providing application, an online video rendering application, a live sports event streaming application, a live video conferencing application, etc.

[0045] In one example, for an end-to-end ASR system 104, the encoder 201 is an acoustic encoder and the decoder 202 is an attention-based decoder. The acoustic encoder processes the input acoustic signal 103 and generates a sequence of encoder states that provide an alternative, e.g., higher-level, representation of the input acoustic signal 103. The sequence of encoder states may include an alternative sequence of frames of audio data corresponding to a second set of time steps. In some implementations, the alternative representation of the input acoustic sequence is subsampled to a lower frame rate, i.e., the second set of time steps in the alternative representation are smaller than the first set of time steps in the input acoustic sequence. The attention-based decoder is trained to process the encoder states that represent the alternative representation of the input acoustic signal 103 and generate a transcription output from the sequence of encoder states that are provided to the attention-based decoder.

[0046] Some embodiments are based on the recognition that an attention-based ASR system may need to observe an entire speech utterance segmented by speech pauses and assign a weight to each input frame in order to recognize each transcription output 203. Without a priori knowledge of which parts of the input acoustic signal are relevant to recognizing the next transcription output, and because a weight must be assigned to each encoder state, attention-based decoders typically need to process large input sequences. While such processing allows taking advantage of attention to different parts of an utterance, it also increases output delay and is therefore impractical for streaming / online speech recognition.

[0047] Some embodiments are based on the recognition that an example of a priori knowledge regarding the relevance of different portions of an input sequence to the recognition of a next transcription output is an indication of the position of the frame in the input sequence that corresponds to the transcription output to be recognized. Indeed, if the transcription output positions are known, an attentional decoder can be forced to pay more attention to these positions and less attention to other positions by restricting the input sequence. In this way, for each transcription output, the attentional network can focus its attention on its position in the input sequence. This guided attention reduces the need to process large input sequences, which in turn reduces output latency, making attentional decoders practical for streaming / online recognition.

[0048] To this end, the decoder 202 is an alignment decoder trained to determine the position of an encoder state in a sequence of coded states that encodes a transcription output, such as a character, bit, or word. For example, connectionist time series classification (CTC) is a type of objective function and associated neural network output for training a recurrent neural network (RNN), such as a long short-term memory (LSTM) network, to address timing-variable sequence problems. A CTC-based ASR system is an alternative to an attention-based ASR system. A CTC-based neural network generates an output for each frame of an input sequence; i.e., the input and output are synchronized, and a beam search algorithm is used to find the best output sequence before collapsing the neural network output into an output transcription. The performance of an attention-based ASR system may be superior to that of a CTC-based ASR system. However, some embodiments are based on the recognition that the input / output frame alignment used by intermediate operations of a CTC-based ASR system can be used by an attention-based ASR system to address the aforementioned drawback of its output delay.

[0049] 3 shows an example architecture of such a CTC-based ASR system 300, where the encoder 201 is a self-attention encoder 301. The CTC-based ASR system 300 also includes an attentional decoder 303.

[0050] The encoder 301 processes the input acoustic signal 103 and generates a sequence of encoder states 302 that provide an alternative, e.g., higher-level, representation of the input acoustic signal 103. The sequence of encoder states 302 may include an alternative sequence of frames of audio data corresponding to a second set of time steps. The attention-based decoder 303 is trained to process the sequence of encoder states 302 that represent the alternative representations of the input acoustic signal 103 and to generate a transcription output 304 (corresponding to output 203) from the sequence of encoder states provided to the attention-based decoder 303.

[0051] The CTC-based ASR system 300 also includes a decoder 202 that is an alignment decoder 305 for utilizing alignment information 306. The alignment information 306 includes an alignment of sequences of transcription outputs annotated with speaker identities in the multi-speaker input acoustic signal 103. Such a CTC-based ASR system 300 includes a partition module 307 configured to partition a sequence of encoder states 302 into a set of partitions 308. For example, the partition module 307 can partition the sequence of encoder states for each position 306 of an identified encoder state, where the number of partitions 308 is equal to, for example, the number of identified encoder states 302 that encode the transcription output. In this manner, the attention-based decoder 303 accepts as input not the entire sequence 302, but portions 308 of the sequence, each portion likely containing a new transcription output for forming the transcription output sequence 304. In some implementations, the combination of the alignment decoder 305, the attention-based decoder 303, and the partition module 307 is referred to as a triggered attention decoder. In effect, the triggered attention decoder processes portions of an utterance as it is received, making the CTC-based ASR system 300 practical for recognition in a streaming / online manner.

[0052] In some prior art end-to-end ASR systems, label sequences for different speakers are assumed to be output in different output heads, or prediction of a sequence for a speaker can only begin once the sequence of the previous speaker has been completed.

[0053] However, in the end-to-end ASR system 104 disclosed in various embodiments provided herein, the multi-speaker ASR problem is not implicitly viewed as a source separation problem using separate output layers for each speaker or a cascaded process to recognize each speaker in turn. Instead, the prediction of ASR labels for multiple speakers is viewed as a sequence of acoustic events regardless of their source.

[0054] To that end, some embodiments use a generalized form of CTC previously disclosed in GTC to achieve multi-speaker separation using an extended GTC (GTC-e) loss.

[0055] FIG. 4 shows a schematic diagram 400 illustrating an extension of the GTC 401 objective function for the end-to-end ASR system 104 to perform multi-speaker separation, according to some embodiments.

[0056] The objective function of GTC401 is extended to yield the GTC-e402 loss, which allows training two separate predictions, one for the speakers (e.g., speakers 101 and 102) and one for the ASR output (e.g., output 203), aligned at the frame level for the end-to-end ASR system 104. To efficiently utilize the speaker predictions during decoding, the existing frame-synchronous beam search algorithm of GTC401 is adapted to GTC-e402.

[0057] The objective function of GTC 401 provides an output in the form of a directed graph 403, where nodes represent labels and edges represent label transitions. Meanwhile, the objective function of GTC-e 402 provides as output a directed graph 404, where nodes represent tokens and edges represent speaker identities (IDs). Thus, the objective function of GTC-e 402 is configured to perform multi-speaker ASR by considering the ASR output of multiple speakers as a mixed sequence of events with a meaningful chronological ordering.

[0058] To do so, the objective function of GTC-e402 is used as the loss function of a neural network, which is trained to receive an input sequence of labels corresponding to multiple speakers and provide as output chronologically separated labels and speaker identification for each label.

[0059] Figure 5 shows the architecture of an end-to-end ASR system 104 that uses a neural network 501 trained with the objective function of GTC-e 402. The neural network 501 includes the encoder 201 and decoder 202 described in Figure 2. The neural network 501 is trained to achieve multiple objectives of speech recognition and speaker identification.

[0060] In various embodiments, the encoder 201 is a multi-head encoder, and the decoder 202 is a time-aligned decoder (as shown in FIG. 3). The multi-head encoder and the time-aligned decoder are part of a neural network 501 that is trained end-to-end to recognize and / or transcribe each speaker's utterance. To do so, the neural network 501 is trained to achieve multiple objectives, namely, speech recognition and speech identification. To achieve this training, in some implementations, the neural network 501 is trained using multiple loss functions, one for speech recognition and another for speaker identification. Doing so simplifies the construction of the loss functions and / or enables the reuse of legacy loss functions designed for different applications.

[0061] To that end, neural network 501 is trained to minimize a loss function that includes a first component associated with errors in speech recognition and a second component associated with errors in speaker identification.

[0062] However, employing multiple loss functions or multiple components of loss functions can cause synchronization issues between the outputs of different heads of the multi-head encoder. This is because there is no alignment information between the acoustic frames and labels of the input acoustic signal 103, and separately aligning each transcription with each speaker ID information results in misalignment. Therefore, some embodiments train the multi-head encoder 201 of the neural network 501 using a single loss function configured to simultaneously minimize errors in speech recognition and speaker identification.

[0063] Some embodiments recognize that training of the end-to-end ASR system 104 can be performed using a CTC objective that allows the decoder 202 to perform alignment. For example, in speech, there can be multiple time slices corresponding to a single phoneme. Because the alignment of the observed sequence with the target label is unknown, training with a CTC objective predicts a probability distribution at each time step.

[0064] The CTC objective uses a graph-based loss function to train a neural network 501 when there is no temporal alignment information available between a sequence of training labels and a longer sequence of label probabilities output by the neural network 501, which is computed from a sequence of observations input to the neural network 501. Such missing temporal alignment information creates a temporal ambiguity between the sequence of label probabilities output by the neural network 501 and the supervised information for training, which is a sequence of training labels, that can be resolved using the CTC objective function.

[0065] However, the CTC objective function is only suitable for resolving temporal ambiguity during neural network training. If other types of ambiguity need to be considered, the CTC objective function will not be useful. Therefore, an objective of some embodiments is to enhance the CTC objective function to consider other ambiguities, such as speaker identification.

[0066] Some embodiments are based on the recognition that, although the definition of the CTC objective and / or the CTC rules is not graph-based, problems or limitations of the CTC objective can be represented by a directed graph and solved using a graph-based definition. Specifically, if the CTC rules cause supervision information of a training label sequence to reside on a graph that implements an alignment between a sequence of label probabilities generated by a neural network and a sequence of training labels, it would be advantageous to extend this graph principle to address speaker alignment.

[0067] In one example, the neural network 501 is trained using an augmented CTC objective function. As GTC is known to be a generalized form of the CTC objective function, therefore, in one embodiment, the neural network 501 is trained using a GTC-e402 objective function, also referred to as the GTC-e402 loss function. The GTC-e402 objective function (or the augmented CTC objective function) is used to perform alignment between inputs and outputs on a graph with nodes representing speech identification outputs, also known as transcription outputs. The edges of the graph represent transitions between multiple speakers. Such a graph is shown in FIG. 6.

[0068] FIG. 6 illustrates an example operation 600 of neural network 501 according to an exemplary embodiment. Example operation 600 illustrates a graph 602 having a plurality of nodes and edges. For example, node 603 is represented by the text "Hello," and edge 604 is represented by the text "s1." In graph 602, each node represents a label, and an edge connecting two nodes represents the likelihood of a transition between those two nodes. Some embodiments are based on the understanding that one way to address speaker ambiguity is to annotate nodes and edges with not only labels but also speaker identities. Thus, in graph 602, nodes such as node 603 and node 605 are associated with labels that indicate ASR outputs. For example, node 603 is associated with the label "Hello," node 605 indicates the starting node, edge 604 indicates the speaker with identity s1, and edge 606 indicates the speaker with identity s2. Similarly, other nodes and edges in graph 602 are annotated, although not all of the annotations are shown for the sake of brevity and do not limit the scope of the present disclosure.

[0069] Additionally or alternatively, some embodiments are based on the understanding that for every ASR output, in the form of labels on nodes, speaker labels are predicted at the frame level, in the form of annotations on edges, in graph 602. Speaker information can be viewed as transition probabilities in graph 602, and such annotations enable synchronization of speaker and ASR label predictions at the frame level.

[0070] As shown in FIG. 6, a multi-speaker overlapping speech input acoustic signal 103 is received by a neural network 501. For simplicity, the overlapping speech input acoustic signal 103 is formed by overlapping speech from two speakers s1 and s2 (shown as first speaker 101 and second speaker 102, respectively, in FIG. 1). Speaker s1 utters "Hello Cat," and speaker s2 utters "Hi Dog." The neural network 501 processes the overlapping speech input acoustic signal 103 using an encoder 201 and a decoder 202 and based on the objective function of the extended CTC objective function, GTC-e402. As a result of the processing, a graph 602 is obtained in which the nodes of the graph indicate the transcription outputs corresponding to the utterances "Hello," "Hi," "Cat," and "Dog" in time series, and the edges provide the corresponding speaker IDs s1, s2, s1, and s2 in time series. Thus, the transcription output 107 from the neural network 501 includes both a synchronized label output 107a and a synchronized speaker identification output 107b. This synchronization occurs for each frame of the input acoustic signal 103.

[0071] In one embodiment, the objective function of GTC-e 402 uses supervision information from a directed graph of nodes connected by edges representing labels and transitions between labels, the directed graph representing possible alignment paths for the sequences of probability distributions and labels output by neural network 501. A description of the objective function of GTC-e 402 is covered in the following description.

[0072] To understand the principles of the GTC-e402 objective function, it is first necessary to understand the principles of the GTC objective function.

[0073] 7A is a schematic diagram illustrating a workflow for training a neural network 701 using a graph-based time series classification (GTC) objective function 702, according to an example embodiment. The neural network 701 is trained to output a sequence of probability distributions 703 for a sequence of observations 705, where the sequence of probability distributions 703 indicates the label probabilities at each time point. The type of sequence of observations 705 and sequence of labels 706a input to the neural network 701 depends on the type of application in which the neural network 701 is being used.

[0074] For example, in the case of a neural network 701 associated with an ASR system, the sequence of observations 705 provided at the input interface of the neural network 701 may be associated with a speech utterance, and the sequence of labels 706a may correspond to words, subwords, and / or characters from the alphabet of a particular language. Furthermore, in an acoustic event detection application, if the neural network 701 is trained to detect different acoustic events occurring in a particular time span within an acoustic scene, the sequence of observations 705 may include different acoustic features of sounds included in the particular time span within the acoustic scene. In this case, the sequence of labels 706a may include labels corresponding to different entities that generate the sounds or cause the acoustic events. For example, for a sound such as a cat meowing in an acoustic scene, the label "cat sound" may be used, and similarly, for a barking sound, the label "dog sound" may be used. Thus, the sequence of observations 705 and the sequence of labels 706a will vary depending on the application.

[0075] Neural network 701 is trained using a GTC objective function 702, which uses supervision information from a directed graph 704. The directed graph 704 includes multiple nodes connected by edges, which represent labels and transitions between labels. Some embodiments recognize that presenting supervision information on the directed graph 704 allows different rules to be applied to train the neural network in a manner consistent with the principles of such training. This is because the structure of the directed graph 704 is consistent with the differentiable approach used by the forward-backward algorithm for training. Thus, when rules desired to be imposed for training are represented as part of the structure of the directed graph 704, such rules can be imposed for training in a differentiable manner consistent with the forward-backward algorithm.

[0076] For example, in one embodiment, directed graph 704 represents multiple possible alignment paths for sequence of probability distributions 703 and sequence of multiple labels 706a. Such a directed graph enables neural network 701 to be trained using the GTC objective to perform alignment between its inputs and outputs in both the time and label domains. To achieve this multi-alignment, the structure of directed graph 704 is non-monotonic, i.e., it specifies non-monotonic alignments between a sequence of labels in sequence of multiple labels 706a and sequence of probability distributions 703.

[0077] Additionally or alternatively, in one embodiment, the directed graph 704 represents a constraint 706b on label repetition. The constraint 706b on label repetition specifies a minimum number of label repetitions, a maximum number of label repetitions, or both. The constraint 706b on label repetition reduces the number of possible label sequences that may be generated during the evolution of the directed graph 704 for time alignment, which can speed up the computation of the GTC loss.

[0078] The sequence of observations 705 may correspond to features extracted by a feature extraction method. For example, the observations may be obtained by dividing an input signal into overlapping chunks and extracting features from each of the chunks. The type of feature extracted may vary depending on the type of input. For example, for a speech utterance, features extracted from a chunked sequence of speech samples may include spectral decomposition of the input signal and additional signal processing steps to mimic the frequency resolution of the human ear. For example, each feature frame extracted from the input speech utterance may correspond to a time instance in the sequence of observations 705, e.g., each frame of the speech utterance is associated with a 25-millisecond speech sample that is shifted in time by an additional 10 milliseconds from the previous frame of the speech utterance. Each feature frame of a speech utterance in the sequence of feature frames of the speech utterance may include acoustic information characterizing the portion of the utterance at the corresponding time step. For example, the sequence of feature frames of speech data may include a filterbank spectral energy vector. Neural network inputs and outputs

[0079] In various embodiments, the input to neural network 701 is a sequence of observations 705, and the output of neural network 701 is a sequence of probability distributions (also known as likelihoods) 703 over a set of labels. For clarity of explanation, the probability distributions 703 generated by neural network 701 are described below using an exemplary embodiment in which neural network 701 is trained for automatic speech recognition (ASR). However, this example is not intended to limit the scope, applicability, or configuration of embodiments of the present disclosure.

[0080] Figure 7B shows a sequence of probability distributions 703 computed from a sequence of observations 705 by a neural network 701 trained for ASR, according to an exemplary embodiment. Figure 7B will be described in conjunction with Figure 7A. The input to the neural network 701 includes the sequence of observations 705 along with features extracted from speech utterances. The neural network 701 is trained based on supervision information that includes a directed graph 704 that encodes possible speech recognitions with some ambiguity.

[0081] The directed graph 704 and sequence of probability distributions 703 are processed by a GTC objective function 702 to optimize the temporal and label alignment of labels in the directed graph 704 with respect to the input sequence of observations, and to find gradients for updating the parameters of the neural network 701. The neural network 701, trained using the GTC objective function 702, generates a matrix of sequence of probabilities 703, where the columns correspond to time steps and each row corresponds to a label (here, a letter of the English alphabet).

[0082] In the example of FIG. 7B, neural network 701 outputs a D×T dimensional matrix (D represents the label dimension and T represents the time dimension, with D=29 and T=30 in the given example) or sequence of probability distributions 703, where letters of the English alphabet and some special characters correspond to D=29 labels. Each column (D dimension) of the D×T matrix corresponds to a probability that sums to 1, i.e., the matrix shows a probability distribution over all labels for each time step. In this example, the labels correspond to characters of the English alphabet from A to Z plus the additional symbols "_", ">", and "-", where "-" indicates a blank token or whitespace symbol. The sequence of probability distributions 703 defines, for each time step, the probabilities of different labels, which are calculated by neural network 701 from the sequence of observations 705. For example, as observed in Figure 7B, the probability of observing the label "B" at the fourth time step is 96%, the probability of the label "O" is 3%, and the probabilities of the remaining labels are near zero. Thus, the most likely sequence of labels in the output of this example has either the letter "B" or "O" at the fourth time position. During inference, the final sequence of labels may be extracted from the sequence of probability distributions 703 using various techniques, such as prefix beam search over the labels.

[0083] Furthermore, by using the GTC objective, the neural network 701 is trained to maximize the probability of a sequence of labels in a sequence of probability distributions 703 corresponding to a sequence of nodes and edges included in the directed graph 704. For example, suppose a ground truth transcription of an input speech utterance corresponds to “BUGS_BUNNY,” but the ground truth transcription is unknown. In such a case, the directed graph 704 can be generated from a list of ASR hypotheses for the speech utterance corresponding to “BUGS_BUNNY.” For example, the list of ASR hypotheses represented by the directed graph 704 can be “BOX_BUNNY,” “BUGS_BUNNI,” “BOG_BUNNY,” etc. (where each letter of the English alphabet corresponds to a label). Because it is unknown whether any of the hypotheses are correct or which parts of the hypotheses are correct, such a list of multiple hypotheses for the speech utterance corresponding to “BUGS_BUNNY” includes ambiguous label information, unlike the ground truth information for “BUGS_BUNNY.”

[0084] During GTC training, the directed graph 704 is unfolded to the length of the sequence of probability distributions 703, and each path from a particular start node to a particular end node in the unfolded graph represents an aligned path and a label sequence. Such a graph may contain non-monotonic alignments between the sequence of probability distributions 703 output by the neural network 701 and the sequence of labels 706a encoded in the graph. One of the aligned paths contained by the directed graph 704 may correspond to the sequence of labels: "-BOOXXX_BBUUN-NI," "B-OOX-_-BUNN-NY-," "BUGS_BUNNY-," etc. (- indicates a space symbol). Each sequence of labels in the directed graph 704 includes a temporal alignment and a label alignment. The temporal and label alignments of the sequence of labels and probability distributions 703 in the directed graph 704 are optimized by processing the directed graph 704 and training the neural network 701 using the GTC objective function 702. The GTC objective function 702 is used to train the neural network 701 to maximize the probability of a sequence of labels contained in the directed graph 704. Transition weights present on the edges of the directed graph 704 may be used during training to emphasize more likely alignment paths. To that end, in an exemplary embodiment, each hypothesis may be assigned a score by the neural network 701. Furthermore, each hypothesis may be ranked based on its score. Furthermore, based on the ranking, weights may be assigned to transitions corresponding to each hypothesis, such that the weight of a transition corresponding to a first ranked hypothesis is greater than the weight of transitions corresponding to subsequent hypotheses in the N-best hypothesis. For example, a hypothesis "BOG" may have a higher rank compared to another hypothesis "BOX" based on contextual information. Thus, the weight connecting the labels "O" and "G" may be greater than the weight of the connection between "O" and "X." Thus, a sequence of labels with larger transition weights will be assigned a higher probability score and, therefore, will be selected to correctly transcribe the input speech utterance. Directed graphs with nonmonotonic ordering

[0085] In some embodiments, supervised information is included in the structure of directed graph 704, and the supervised information is used by GTC objective function 702 to resolve one or more ambiguities, such as temporal ambiguity and label ambiguity, to train neural network 701. Thus, the supervised information specifies one or a combination of non-monotonic alignments between the sequence of labels 706a and the sequence of probability distributions 703. Based on the non-monotonic alignments, directed graph 704 can output multiple unique label sequences.

[0086] 7C illustrates an exemplary directed graph 700c according to an exemplary embodiment. Directed graph 700c includes multiple nodes 707a, 707b, 707c, and 707d, each representing a label. For example, node 707a represents label "A," node 707b represents label "B," node 707c represents label "C," and node 707d represents label "D." Directed graph 700c begins with start node 711a and ends with end node 711b. In FIG. 7C, the start and end nodes are connected to the labels with dashed lines to indicate that there may be other nodes in directed graph 700c that are not shown for brevity and clarity.

[0087] Directed graph 700c is a non-monotonic directed graph, thereby providing a non-monotonic alignment between the sequence of labels of directed graph 700c and the sequence 703 of probability distributions output by neural network 705 during training. In different embodiments, the non-monotonic alignment may be implemented differently to allow multiple passes through the nodes of directed graph 700c, capturing label and time ambiguity.

[0088] For example, as shown in Figure 7C, a non-monotonic ordering in directed graph 700c can be constructed by connecting at least one node to a different node that represents a different label. For example, node 707a, representing label A, is connected by edge 709ab to node 707b, representing label B, and also by edge 709ac to node 707c, representing label C. Such split connections make it possible to create multiple sequences of different labels defined by multiple different paths through the graph, such as the sequence ABD and the sequence ACD sandwiched between the start and end nodes.

[0089] Another example of a non-monotonic ordering encoded in the structure of directed graph 700c is a loop formed by an edge connecting multiple non-blank nodes. In directed graph 700c, the loop is formed by edges 709ab and 709ba, making it possible to create multiple paths through the graph, such as ABACD or ABABD.

[0090] Some embodiments are based on the recognition that because the non-monotonic directed graph 700c encodes sequences of different labels, not all sequences are equally likely. Therefore, it is necessary to impose unequal probabilities on the structure of the directed graph 700c.

[0091] A further advantage of directed graph 700c is its ability to encode transition probabilities as edge weights, which in turn encode the probabilities of different paths. To that end, at least some edges in a non-monotonic directed graph are associated with different weights (w), making directed graph 700c a weighted directed graph 700c. For example, edge 709ab may be weighted with weight w2, edge 709ba may be weighted with weight w1, edge 709bd may be weighted with weight w3, edge 709ac may be weighted with weight w4, and edge 709cd may be weighted with weight w5. Furthermore, the conditional probability of a sequence of nodes may be modified based on the weights. For example, if weight w2 is greater than weight w1, then in a particular sequence of nodes, the conditional probability of a transition from node 707a to node 707b is greater than the conditional probability of a transition from node 707b to node 707a. Directed graphs with constraints on label repetition

[0092] FIG. 7D illustrates the repetition of labels during the expansion of a directed graph 700d, according to an exemplary embodiment. FIG. 7D includes directed graph 700d on the left and expanded directed graph 710d on the right. Directed graph 700d includes a sequence of labels corresponding to the transcription "HELLO WORLD." Assume that there are more observations in the sequence of observations 705 provided to neural network 701 than there are labels, i.e., transcriptions, in the sequence of labels. For example, the number of characters in the transcription "HELLO WORLD" may be 10, and the number of observations (and corresponding conditional probabilities) may be 30. Therefore, to match or align the number of labels with the number of observations, some labels in the transcription are repeated during the expansion of the graph. For example, the letter "E" in the transcription "HELLO WORLD" may be repeated several times.

[0093] However, due to the lack of constraints on the number of times a label can be repeated, the GTC objective function is required to analyze the possible transitions from each repeated label, resulting in unnecessary waste of computational power. To that end, directed graph 700d includes constraints 706b on label repetition. Constraints 706b in directed graph 700d may include a minimum number of times a label is allowed to repeat in a sequence of labels, a maximum number of times a label is allowed to repeat in a sequence of labels, or both. This is because the letter "E" is unlikely to be observed over so many consecutive time frames as in example expansion 712.

[0094] Therefore, in addition to or as an alternative to non-monotonic alignment of directed graph 700d, some embodiments use the structure of directed graph 700d to impose constraints on label repetition during training, specifying a minimum number of label repetitions, a maximum number of label repetitions, or both. Such constraints on label repetition for nodes representing labels can be achieved by removing self-transitions for the node and adding transitions to other nodes that represent the same label.

[0095] 7E shows an example directed graph 700e with a constraint 706b on label repetition, according to an example embodiment. Directed graph 700e begins at start node 713a and ends at end node 713b. Monotonic directed graph 700e includes multiple nodes 714x, 715y, 714y, and 714z, each representing a label. For example, node 714x represents label "X," 714y represents label "Y," node 714z represents label "Z," and 715y represents another label "Y." In this example, a sequence of connected nodes representing the same label is formed by nodes 714y and 715y.

[0096] Directed graph 700e is monotonic because although there are multiple paths through the nodes of the graph connecting the start node and the end node, after the folding procedure only a single sequence of labels XYZ can be formed.

[0097] For example, monotonic directed graph 700e may specify different sequences of labels, such as X→X→X→Y→Z→Z→ or X→Y→Y→Z or X→Y→Z, during the unfolding of monotonic directed graph 700e. However, after collapsing these label sequences, only one label sequence is produced: X→Y→Z. In some embodiments, multiple monotonic directed graphs can be combined to form a non-monotonic directed graph (such as non-monotonic directed graph 700c) for use in training neural network 701.

[0098] Furthermore, in monotonic directed graph 700e, it can be defined that certain labels, e.g., label “Y,” must not be repeated more than twice, while labels “X” and “Z” can be repeated multiple times. This information is encoded in the structure of the graph and used in an automatic manner during expansion. For example, nodes 714x and 714z have self-transitions and can therefore be repeated any number of times allowed by the expansion. In contrast, nodes 714y and 715y, corresponding to label “Y,” do not have self-transitions. Thus, to traverse the graph between the start and end nodes, the path could be 714x-714y-714z, where label “Y” corresponding to node 714y is repeated once, or 714x-714y-715y-714z, where label “Y” corresponding to nodes 714y and 715y is repeated twice. Furthermore, directed graph 700e allows for modifying or constraining the repetition of other labels, such as labels “X” and “Z,” which are currently repeated multiple times without constraints. Directed graph 700e may be modified to direct graph 700f to impose constraints on other labels "X" and "Z."

[0099] 7F shows another example directed graph 700f with a constraint 706b on label repetition, according to an example embodiment. In FIG. 7F, the structure of monotonic directed graph 700f constrains that label "X" can be repeated only three times in a row, so that node 716x representing label "X" and node 718x, also representing label "X," can be connected to original node 714x. In this example, a sequence of connected nodes representing the same label is formed by nodes 714x, 716x, and 718x.

[0100] Similarly, label "Z" may be constrained to always be repeated twice, etc. To do so, node 717z may be connected to the original node 714z. In this way, directed graph 700f provides great flexibility for optimizing the training of neural network 701.

[0101] The repetition constraint 706b is advantageous for speech-related applications. For example, for a directed graph 700f used by a neural network 701 corresponding to an ASR system configured to transcribe in English, it may be known in advance that the output corresponding to the label "U" is unlikely to be observed across multiple consecutive frames. Therefore, the label "U" may be constrained to be repeated only a limited number of times to reduce computational complexity and speed up computation for GTC purposes.

[0102] The benefits of the constraint 706b on repetition are not limited to speech-related applications. For example, the directed graph 700f and neural network 701 can correspond to an acoustic event detection system implemented to detect acoustic events in a residential environment. A short event such as a "door slam" may be unlikely to occur over many consecutive observation frames. Thus, the structure of the directed graph 700f may define a constraint 706b on repetition of the label "door slam." Training for GTC purposes using directed graphs

[0103] In various embodiments, a neural network 701 is trained based on a GTC objective function 702 to convert a sequence of observations 705 into a sequence of probability distributions 703. The neural network 701 is further configured to unfold a directed graph 704 to generate all possible sequences of labels from a plurality of sequences of labels 706a such that the length of the sequence of labels matches the length of the sequence of probability distributions 703. Unfolding the directed graph 704 includes generating sequences of labels and ordered paths by finding paths from start nodes to end nodes through the nodes and edges of the directed graph 704 according to the structure of the directed graph 704, the paths being the length of the sequence of probability distributions 703. Each path in the unfolded graph corresponds to a fixed-length sequence of nodes and edges starting at a specific start node and ending at a specific end node. Each possible path in the unfolded graph, corresponding to a sequence of nodes and edges, can be mapped to a sequence of labels.

[0104] Additionally, neural network 701 updates one or more parameters of neural network 701 based on a GTC objective function 702 configured to maximize the sum of the conditional probabilities of all possible sequences of labels 706a generated by unfolding directed graph 704. The one or more parameters of neural network 701 updated by neural network 701 may include neural network weights and biases, as well as other trainable parameters such as embedding vectors.

[0105] In some embodiments, directed graph 704 is a weighted graph in which at least some edges are associated with different weights. Furthermore, GTC objective function 702 is configured to learn temporal and label alignment to obtain an optimal pseudo-label sequence from weighted directed graph 704, and to update neural network 701 such that training neural network 701 using GTC function 702 reduces the loss associated with the optimal pseudo-label sequence. Neural network 701 trained using GTC objective function 702 converts a sequence of observations 705 into a sequence of probability distributions 703 over all possible labels at each time point. Furthermore, trained neural network 701 maximizes the probability of a sequence of labels corresponding to a sequence of nodes and edges present in directed graph 704 at the output of neural network 701.

[0106] Thus, the GTC objective function 702 enables the neural network 701 to learn and update its parameters with label information in graph form.

[0107] The directed graph 704 provides supervision information used by the GTC objective function 702 in training the neural network 701. In the directed graph 704, a sequence of labels is represented by multiple nodes and edges. Furthermore, the directed graph 704 may include non-monotonic alignments between the sequence of probability distributions 703 and the sequence of multiple labels 706a represented by the directed graph 704. A non-monotonic alignment or a monotonic alignment is defined as the number of label sequences that can be generated from the directed graph 704 by transitioning from a particular start node to a particular end node after removing label repetitions and blank labels. A non-monotonic alignment would allow the directed graph 704 to output multiple unique label sequences, while a monotonic graph would only allow a single label sequence to be output.

[0108] Due to the non-monotonic alignment feature, directed graph 704 contains information associated not only with variations in the sequence of labels in the time domain, but also with variations in the sequence of labels in the label domain itself. Due to variations in the sequence of labels in the label domain, directed graph 704 includes multiple paths through the nodes and edges of directed graph 704, each path corresponding to at least one sequence of labels in the multiple sequences of labels 706a. Thus, each edge in directed graph 704 has a direction from one node to another.

[0109] Thus, the unaligned features allow the directed graph 704 to consider different sequences of labels during training, which allows training of the neural network 701 with ambiguous label information to account for uncertainty about the correct transcription of the training samples.

[0110] Additionally, the directed graph 704 allows repetition of at least one label of the sequence of labels for a certain minimum number of times and a certain maximum number of times during the unfolding of the directed graph 704 to reduce the number of possible label paths that can be generated from the unfolded graph and speed up the computation of the GTC loss.

[0111] In some embodiments, the non-monotonic directed graph 704 is a weighted graph in which at least some edges are associated with different weights. Furthermore, conditional probabilities of sequences of nodes may be calculated during training based on the weights of corresponding edges in the directed graph 704.

[0112]

number

[0113]

number

[0114]

number

[0115]

number

[0116]

number

[0117]

number

[0118]

number

[0119]

number

[0120] The neural network 701 is trained using a GTC objective function 702 that enables the neural network 801 to resolve temporal alignment or temporal ambiguity as well as label alignment or label ambiguity to learn the optimal alignment between the sequence of probability distributions 703 and the sequence of labels represented by the directed graph 704.

[0121]

number

[0122]

number

[0123]

number

[0124] In step 807, the gradients of the neural network parameters are calculated using the derivatives of the GTC objective function 702 with respect to all possible output symbols as shown in equations (10) and (4) above, which are efficiently calculated using a forward-backward algorithm and backpropagation. To that end, the forward-backward algorithm finds the forward variable α and the backward variable β, where α and β are mathematically represented in equation (12):

number

[0125] In step 809, the parameters of the neural network 701 may be updated according to the gradients calculated in step 807. To update the parameters, a neural network optimization function may be implemented that defines rules for updating the parameters of the neural network 701. The neural network optimization function may include at least one of stochastic gradient descent (SGD), SGD with momentum, Adam, AdaGrad, AdaDelta, etc.

[0126]

number

[0127] Some embodiments are based on the recognition that the GTC objective function 702 described above needs to be extended to the objective function of GTC-e 402 so that it can be applied to a trained neural network 501 operating in a multi-speaker condition. In the GTC objective function 702, only the posterior probabilities on the nodes were predicted by the neural network 701. However, in the objective function of GTC-e 402, even the weights on the edges of a directed graph such as graph 602 are predicted by the neural network 501. To that end, it has already been discussed that in graph 602, nodes indicate tokens or labels, and edges indicate speaker transitions.

number

[0128]

number

[0129]

number

[0130]

number

[0131]

number

[0132]

number

[0133]

number

[0134]

number

[0135]

number

[0136]

number

[0137]

number

[0138]

number

[0139] Thus, using the above GTC-e402 formulation, a neural network 501 is used to perform speech recognition and speaker separation tasks. Specifically, the neural network 501 may use a different decoder that can perform time alignment of a sequence of likelihoods (or probabilities) of labels or transcription outputs of speech recognition with a sequence of likelihoods of speaker identities. For example, one embodiment extends the principle of suffix beam search to the multi-speaker case. Note that beam search cannot be used in multi-speaker applications that use speech separation as a pre- or post-processing task. However, the introduction of a multi-head encoder makes it possible to adapt suffix beam search to generate a sequence of transcription outputs annotated with speaker identities.

[0140] FIG. 9 illustrates a beam search algorithm used during the decoding operation of neural network 501, according to an example embodiment.

[0141] Because the output of the objective function of GTC-e402 contains tokens from multiple speakers, the existing time-synchronized prefix beam search algorithm is modified as shown in Figure 9. The modification has three main aspects. First, speaker transition probabilities 901 are used in score calculation. Second, all possible speaker IDs are considered (902) to consider all possible speakers when expanding prefixes. Third, subsequences from different speakers are considered separately when calculating the language model (LM) score of the prefix (903).

[0142] These modifications are used by the decoder 202 of the neural network 501 to perform a beam search to generate a chronologically ordered sequence of language tokens, where each token is associated with a speaker identity.

[0143] In some embodiments, LM is employed during inference via shallow fusion, consisting of two long-short-term memory (LSTM) neural network layers, each trained with 1024 units using stochastic gradient descent and LibriSpeech's official LM training text data, excluding sentences occurring in the 860h training data subset. ASR decoding is based on a time-synchronized prefix beam search algorithm. A decoding beam size of 30, a score-based pruning threshold of 14.0, an LM weight of 0.8, and an insertion bonus factor of 2.0 are used.

[0144] FIG. 10 shows Table 1, which shows the greedy search results for the ASR performance of the ASR system 104 based on the objective function of GTC-e 402.

[0145] The word error rates (WER) are shown in Table 1. From the table, it can be observed that the ASR system 104 based on the GTC-e 402 objective function is better than the conventional ASR model. Table 1 shows the WER for three models: a single-speaker CTC model 1001, a PIT-CTC model 1002, and a GTC-e model 1003. The GTC-e model 1003 is a GTC-e 402-based ASR system 104 disclosed in various embodiments described herein. The GTC-e model achieves performance close to the PIT-CTC model 1002, especially in the case of low overlap rates (0%, 20%, and 40%) 1004.

[0146] FIG. 11 shows Table 2, which shows the greedy search results for the ASR performance of the ASR system 104 based on the objective function of GTC-e 402 based on the oracle token error rate.

[0147] In Table 2, the oracle TER for PIT-CTC 1002 and GTC-e model 1003 is shown to simply compare tokens from all output sequences with all reference sequences, regardless of speaker assignment.

[0148] The averaged test TERs for PIT-CTC1101 and GTC-e1102 were 22.8% and 25.0%, respectively, establishing comparable token recognition performance.

[0149] GTC-e1003 is able to accurately predict the activation of most tokens, which is a very good performance indicator.

[0150] FIG. 12 shows Table 3 illustrating the beam search results for the ASR performance of the ASR system 104 based on the objective function of GTC-e 402 based on WER.

[0151] For the beam search decoding results in Table 3, a 16-layer Transformer-based LM for the language model was trained on the full LibriSpeech data using external text. The beam size for GTC-e1003 is set to 40, and the beam size for PIT-CTC1002 is cut in half to keep the averaged beam size for all speakers the same. Beam search significantly improves the word error rate.

[0152] 13 shows Table 4 illustrating beam search results for the ASR performance of ASR system 104 based on the objective function of GTC-e 402 based on WER for multiple speakers. The rows of Table 4 show the WER for different overlap cases for GTC-e2 speaker model 1301 (e.g., the objective function of GTC-e 402 of ASR system 104), speaker 1 1302, and speaker 2 1303.

[0153] From the table, we can see that the GTC-e model 1301 is not biased towards any speaker and gives the same WER for each speaker.

[0154] Based on the performance results, it can be established that the objective function of GTC-e402 is advantageous for multi-speaker separation and speech recognition tasks with good performance. Therefore, the objective function of GTC-e402 can be used in various neural network architectures to perform end-to-end ASR.

[0155] FIG. 14A illustrates a neural network architecture 1400a of an ASR system implemented using the objective function of GTC-e 402, according to an example embodiment.

[0156] In some embodiments, the neural network architecture 1400a corresponds to a transducer-based neural network architecture that employs the proposed GTC-T loss function to train a neural network (e.g., neural network 501).

[0157]

number

[0158]

number

[0159]

number

[0160]

number

[0161]

number

[0162] FIG. 14A shows a neural network architecture 1400a of an ASR system implemented using a GTC-T objective function.

[0163] In some embodiments, neural network architecture 1400a corresponds to a transducer-based neural network architecture that employs the proposed GTC-T loss function 1401 to train a neural network (e.g., neural network 501), where GTC-T loss function 1401 corresponds to the GTC-T objective function. In neural network architecture 1400a, 80-dimensional log-mel spectral energy plus three additional features for pitch information are used as input as acoustic features, which are derived from audio input 1402 using feature extraction module 1403.

[0164] In some embodiments, the neural network architecture 1400a consists of a two-layer convolutional neural network (CNN) model 1405 followed by a stack of an E=12 transformer-based encoder layer 1406, a linear layer 1407, a prediction network 1408, a combination network 1409, and a final softmax function 1410, which maps the neural network output to a posterior probability distribution. In some exemplary embodiments, each layer of the two-layer CNN model 1405 may use a stride of 2, a 3x3 kernel size, 320 channels, and a rectified linear unit (ReLU) activation function. Furthermore, a linear neural network layer 1407 is applied to the output of the last CNN layer. A sinusoidal position encoding 1411 is added to the output of the two-layer CNN model 1405, which then feeds it to the transformer-based encoder 1406. Each transformer layer employs a 320-dimensional multi-head self-attention layer with four attention heads, layer normalization, and an internal-dimensional feedforward neural network module 1540. The remaining connections are applied to the outputs of the multi-head self-attention and feedforward modules.

[0165] In one embodiment, the HKUST and LibriSpeechASR benchmarks are used for evaluation. HKUST is a corpus of Mandarin telephone speech recordings with over 180 hours of transcribed speech data, while LibriSpeech contains approximately 1k hours of spoken English audiobooks. In one example, the ASR system is configured to first extract 80-dimensional log-melt spectral energy plus three additional features for pitch information. The derived feature sequence is processed by a VGG neural network, which downsamples the feature sequence to a 40 ms frame rate before being fed to the encoder 1406. The encoder 1406 consists of 12 conformer blocks, each of which includes a self-attention layer, a convolutional module, and two macaron-shaped feedforward neural network modules. In addition, the input to each component of the conformer block is layer-normalized, and dropout is applied to the output of some neural network layers.

[0166] The hyperparameters of the encoder 1406 are d for HKUST. model =256, d=2048, d h = 4, and E = 12, but for LibriSpeech, d model and d hare increased to 512 and 8, respectively. For the CTC model, the output of the encoder neural network is projected onto the number of output labels (including blank labels) using a linear layer and a softmax function to derive a probability distribution over the labels. For the GTC-T and RNN-T loss types, two additional neural network components are used: a prediction network 1408 and a combination network 1409. The prediction network 1408 consists of a single long-short-term memory (LSTM) neural network and a dropout layer. The prediction network 1408 operates like a language model and receives previously emitted ASR labels (ignoring blank labels) as input. The prediction network 1408 transforms the received input of previously emitted ASR labels into an embedding space. The combination network 1409 combines the sequence of encoder frames with the neural network output using a linear layer 1407 and a tanh activation function. Furthermore, a softmax 1410 is used to map the neural network output to a posterior probability distribution. Dropout with a probability of 0.1 is used after the multi-head self-attention and after the feedforward module, as well as for the internal dimension of the feedforward module.

[0167] In some embodiments, SpecAugment-based data augmentation is utilized for training. In one particular example, the ASR output symbols consist of a blank symbol plus 5000 subwords obtained by the SentencePiece method, generated from transcriptions of only a "clean" 100h LibriSpeech training data subset. The ASR model is optimized using the Adam optimizer with β = 0.9, β = 0.98, and ε = 10. -9 The training is performed with a learning rate scheduling with 1.0 and 25,000 warm-up steps. The learning rate factor and the maximum number of training epochs are set to 1.0 and 50 for HKUST and 5.0 and 100 for LibriSpeech.

[0168] In some embodiments, a task-specific LSTM-based language model (LM) is trained and used via shallow fusion during decoding. For HKUST, the LM consists of two LSTM layers with 650 units each. For LibriSpeech, four LSTM layers with 2048 units each are used instead. For LibriSpeech, we also test the effectiveness of a powerful Transformer-based LM (Tr-LM) with 16 layers. The ASR output labels consist of blank tokens plus 5,000 subword units obtained for LibriSpeech, or blank tokens plus 3,653 character-based symbols for the HKUST task.

[0169]

number

[0170] FIG. 14C shows a comparison 1400c of ASR results for CTC, RNN-T, and GTC-T losses on the HKUST benchmark, according to an example embodiment.

[0171] Figure 14C shows ASR results for the CTC, RNN-T, and GTC-T losses on the HKUST benchmark. Joint CTC / RNN-T training and parameter initialization for GTC-T training via CTC pre-training significantly improve the ASR results for both RNN-T and GTC-T-based models. For example, CTC-based initialization only affects the parameters of the encoder 1406, while the parameters of the prediction network 1408 and the combination network 1409 remain randomly initialized. The ASR results demonstrate that for GTC-T training, the use of a CTC-like graph performs better than a MonoRNN-T graph. Additionally, the GTC-T model outperforms the RNN-T model by 0.5% on the HKUST development test set. The use of LM via shallow fusion did not help to significantly improve the word error rate (WER) for RNN-T and GTC-T based ASR models, but it did improve the CTC-based ASR results by between 0.7% and 1.0%. For HKUST, the CTC system also outperformed both the RNN-T and GTC-T systems.

[0172] FIG. 14D shows a comparison 1400d of ASR results for CTC, RNN-T, and GTC-T losses on the LibriSpeech dataset benchmark, according to an exemplary embodiment.

[0173] Figure 14D shows ASR results for the larger LibriSpeech dataset. RNN-T and GTC-T outperform the CTC results. For example, GTC-T with a CTC-like graph, CTC-based initialization, a Transformer-based LM, and a beam size of 30 for decoding achieves a WER of 5.9% for the other test conditions of LibriSpeech. This is 0.9% better than the best CTC result, despite using a powerful LM and a sufficient beam size. The GTC-T result is also 0.3% better than the best RNN-T result. In addition, similar to the HKUST experiment, GTC-T with a CTC-like graph achieves better results than using a MonoRNN-T graph. However, the results in Figure 14D also show that parameter initialization of the encoder 1406 is particularly important for GTC-T training; without initialization, training converges more slowly. For LibriSpeech, the RNN-T model performs better than GTC-T when no external LM is used. Example Implementation

[0174] 15 illustrates a block diagram of a computer-based system 1500 trained using the objective function of GTC-e 402, according to an exemplary embodiment. The computer-based system 1500 may correspond to an end-to-end ASR system 104, an acoustic event detection system, or the like.

[0175] The computer-based system 1500 includes several interfaces that connect the system 1500 to other systems and devices. The system 1500 includes an input interface 1501 configured to accept a sequence of observations 1509, such as a stream of acoustic frames that represent characteristics of an audio utterance. Additionally or alternatively, the computer-based system 1500 can receive the sequence of observations from various other types of input interfaces. In some embodiments, the system 1500 includes an audio interface configured to obtain the sequence of observations 1509 (i.e., a stream of acoustic frames) from an audio input device 1503. For example, the system 1500 may use the sequence of observations 1509, including the acoustic frames, in an ASR application or an acoustic event detection application.

[0176] The input interface 1501 is further configured to obtain, for each sequence of observations in the sequence of observations 1509, a sequence of training labels 1525, where there is no time alignment between the sequence of training labels 1525 and the sequence of probability distributions output by the neural network corresponding to the sequence of observations input to the neural network.

[0177] In some embodiments, the input interface 1501 includes a network interface controller (NIC) 1505 configured to obtain a plurality of sequences of observations 1509 and a plurality of sequences of training labels 1525 over a network 1507, which may be one or a combination of a wired network and a wireless network.

[0178] A network interface controller (NIC) 1505 is adapted to connect the system 1500 to a network 1507 via a bus 1523 and connects the system 1500 to sensing devices, such as input devices 1503. Additionally or alternatively, the system 1500 may include a human-machine interface (HMI) 1511. The human-machine interface 1511 in the system 1500 connects the system 1500 to a keyboard 1513 and a pointing device 1515, which may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, among others.

[0179] System 1500 includes a processor 1521 configured to execute stored instructions 1517 and a memory 1519 that stores instructions executable by processor 1521. Processor 1521 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Memory 1519 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Processor 1521 may be connected to one or more input / output devices via a bus 1523.

[0180] Instructions 1517 may implement a method for training a neural network associated with system 1500 using the objective function of GTC-e402. System 1500 may be used to implement various applications of neural networks, such as end-to-end speech recognition, acoustic event detection, and image recognition, according to some embodiments. To that end, computer memory 1519 stores directed graph 1528, language model 1527, and objective function of GTC-e402. To train system 1500 using the objective function of GTC-e402, directed graph 1528 includes multiple nodes connected by edges, each node representing a label and each edge representing a speaker ID.

[0181] Additionally, paths are generated through the sequence of nodes and edges of the directed graph 1528 that represent the sequence of training labels, and there are multiple paths.

[0182] In some embodiments, the directed graph 1528 is a weighted graph of nodes with associated scores corresponding to the probability that a given transcription output of a given node is the true transcription output at a given time. In some embodiments, transitions from one node to another are weighted, and the weights may be estimated from the scores of the powerful language model (LM) 1527. The directed graph 107 is used by the objective function of the GTC-e 402 to train the system 1500 to convert each sequence of observations 1509 into a sequence of probability distributions over all possible labels at each time by maximizing the probability of a sequence of labels corresponding to the sequence of nodes and edges contained by the directed graph 1528 at the output of the system 1500, and the system 1500 includes an output interface 1535 configured to output the sequence of labels and edges and their likelihoods with respect to the probability distributions.

[0183] In some embodiments, the output interface 1535 may output each probability of the sequence of probability distributions corresponding to each label at each timestamp on a display device 1533. The sequence of probability distributions may be displayed as a matrix. Examples of the display device 1533 include a computer monitor, a television, a projector, or a mobile device, among others. The system 1500 can also be connected to an application interface 1529 adapted to connect the system 1500 to external devices 1531 to perform various tasks, such as sound event detection. Embodiment

[0184] The present description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the claims. In the following description, specific details are provided for a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, like reference numbers and names in the various drawings indicate like elements.

[0185] Also, particular embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a sequential process, many of the operations can be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not discussed or included in the diagram. Moreover, not all operations in any specifically described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to a return of the function to the calling function or the main function.

[0186] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be performed or at least assisted by machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. The necessary tasks may be performed by a processor.

[0187] Furthermore, embodiments of the present disclosure and the functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Furthermore, some embodiments of the present disclosure may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus. Furthermore, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiving device for execution by a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.

[0188] Computer programs (which may also be referred to as or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple cooperating files, such as files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0189] A computer suitable for executing a computer program may include, for example, a general-purpose microprocessor, a special-purpose microprocessor, or both, and any other type of central processing unit, for example, based thereon. Generally, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or be operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer can be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0190] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, that displays information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, that allows the user to provide input to the computer. Other types of devices may also be used to provide user interaction; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including acoustic input, voice (speech) input, or tactile input. Furthermore, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0191] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0192] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0193] Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is therefore the object of the appended claims to cover all such variations and modifications that come within the true spirit and scope of the disclosure.

Claims

1. 1. An end-to-end automatic speech recognition (ASR) system comprising: a processor; and a memory having instructions stored thereon, the processor executing the stored instructions to provide the ASR system with: collecting a sequence of acoustic frames providing a digital representation of an acoustic signal including a mixture of utterances performed by a plurality of speakers; encoding each frame from the sequence of acoustic frames using a multi-head encoder that encodes each frame into a transcription output likelihood and a speaker identity likelihood to generate a sequence of transcription output likelihoods and a sequence of speaker identity likelihoods corresponding to the sequence of acoustic frames; decoding the sequence of transcription output likelihoods and the sequence of speaker identity likelihoods with a decoder that performs alignment to produce a sequence of transcription outputs annotated with the speaker identities; The ASR system submits the sequence of transcription outputs annotated with the speaker identities to a downstream application.

2. The ASR of claim 1 , wherein the decoder uses beam search to generate a chronologically ordered sequence of language tokens, each token associated with a speaker identity.

3. 3. The ASR of claim 2, wherein the beam search is configured to perform operations including one or a combination of: (1) generating speaker transition probabilities and language token probabilities; (2) calculating scores for language tokens; (3) expanding a list of prefixes for all speakers from a set of possible speakers; and (4) calculating scores for prefixes by separately considering subsequences for different speakers.

4. The ASR of claim 1 , wherein the encoder comprises an acoustic encoder configured to process the input acoustic signal and generate a sequence of encoder states, and the decoder comprises an attention-based decoder.

5. 2. The ASR of claim 1, wherein the encoder and decoder form at least a portion of a neural network trained to achieve multiple objective functions by minimizing a loss function including a first component associated with an error in speech recognition and a second component associated with an error in speaker identification.

6. The ASR system of claim 5 , wherein the neural network is trained using a connectionist time series classification (CTC) objective function.

7. 6. The ASR system of claim 5, wherein the encoder and decoder form at least a part of the neural network that is trained with an extended CTC objective function to perform alignment between inputs and outputs on a graph having nodes representing transcription outputs and edges representing speaker transitions.

8. 8. The ASR system of claim 7, wherein the extended CTC objective function is an extended graph-based time series classification (GTC-e) objective function, the GTC-e objective function using supervised information from a directed graph of nodes connected by edges representing labels and transitions between the labels, the directed graph representing possible alignment paths for a sequence of probability distributions output by the neural network and the labels.

9. 9. The ASR system of claim 8, wherein the directed graph represents a plurality of possible alignment paths for the sequence of probability distributions and the sequence of labels, such that possible passages through the structure of the directed graph allow for a plurality of unique label sequences, the plurality of unique label sequences being obtained after collapsing label repetitions and removing blank labels from the plurality of unique label sequences, thereby resulting in a non-monotonic alignment between the sequence of labels and the sequence of probability distributions.

10. 10. The ASR system of claim 9, wherein the non-monotonic alignment is encoded in the structure of the directed graph by allowing transitions from one label to multiple other non-null labels, by allowing transitions from one label to multiple other null labels, or both.

11. The ASR system of claim 7 , wherein the extended CTC objective function is a graph-based time series classification-transducer (GTC-T) objective function.

12. The ASR system of claim 7 , wherein the nodes of the directed graph represent tokens from all speakers in chronological order.

13. The ASR system of claim 7 , wherein the edges of the directed graph indicate speaker identification information.

14. 1. A computer-implemented method for end-to-end automatic speech recognition (ASR), comprising: collecting a sequence of acoustic frames providing a digital representation of an acoustic signal including a mixture of utterances performed by multiple speakers; encoding each frame from the sequence of acoustic frames using a multi-head encoder that encodes each frame into a transcription output likelihood and a speaker identity likelihood to generate a sequence of transcription output likelihoods and a sequence of speaker identity likelihoods corresponding to the sequence of acoustic frames; decoding the sequence of transcription output likelihoods and the sequence of speaker identity likelihoods with a decoder that performs alignment to produce a sequence of transcription outputs annotated with the speaker identities; and submitting the sequence of transcription outputs annotated with the speaker identities to a downstream application.

15. 15. The method of claim 14, wherein the decoder uses beam search to generate a chronologically ordered sequence of language tokens, each token associated with a speaker identity.

16. 16. The method of claim 15, wherein the beam search is configured to perform operations including one or a combination of: (1) generating speaker transition probabilities and language token probabilities; (2) calculating scores for language tokens; (3) expanding a list of prefixes for all speakers from a set of possible speakers; and (4) calculating scores for prefixes by separately considering subsequences for different speakers.

17. The method of claim 14 , wherein the encoder comprises a self-attention encoder and the decoder comprises an attentional decoder.

18. 15. The method of claim 14, wherein the encoder and decoder form at least part of a neural network trained to achieve multiple objective functions by minimizing a loss function including a first component associated with errors in speech recognition and a second component associated with errors in speaker identification.

19. 20. The method of claim 18, wherein the neural network is trained using a connectionist time series classification (CTC) objective function.

20. 20. The method of claim 19, wherein the encoder and decoder form at least part of the neural network that is trained with an extended CTC objective function to perform alignment between inputs and outputs on a graph with nodes representing transcription outputs and edges representing speaker transitions.

Images