Fully supervised speaker logging

Abstract

A method (500) includes receiving a speech utterance (120) and segmenting the speech utterance into a plurality of segments (220). For each segment of the speech utterance, the method further includes extracting a speaker discriminant embedding (240) from the segment and predicting a probability distribution of possible speakers (262) for the segment using a probabilistic generative model (300) configured to receive the extracted speaker discriminant embedding as a feature input. The probabilistic generative model is trained on a corpus of training speech utterances, each training speech utterance being segmented into a plurality of training segments (220T). Each training segment includes a corresponding speaker discriminant embedding and a corresponding speaker label (250). The method further includes assigning a speaker label to each segment of the speech utterance based on the probability distribution of possible speakers for the corresponding segment.

Description

Technical Field

[0001] The present disclosure relates to fully supervised speaker diarization. Background Art

[0002] Speaker logging is the process of dividing an input audio stream into homogeneous segments based on speaker identity. In environments with multiple speakers, speaker logging answers the question of "who spoke when" and has various applications, including multimedia information retrieval, speaker turn analysis, and audio processing, to name a few. In particular, speaker logging systems are able to generate speaker boundaries, which have the potential to significantly improve acoustic speech recognition accuracy. Summary of the invention

[0003] One aspect of the present disclosure provides a method for speaker logging, the method comprising: receiving a speech utterance at data processing hardware and segmenting the speech utterance into a plurality of segments by the data processing hardware. For each segment of the speech utterance, the method further comprises: extracting a speaker discriminant embedding from the segment by the data processing hardware; and predicting a probability distribution of possible speakers about the segment by the data processing hardware using a probabilistic generative model configured to receive the extracted speaker discriminant embedding as a feature input. The probabilistic generative model is trained on a corpus of training speech utterances, wherein each training speech utterance is segmented into a plurality of training segments. Each training segment includes a corresponding speaker discriminant embedding and a corresponding speaker label.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the probability generation model predicts a probability distribution of possible speakers for each segment by applying a distance-dependent Chinese restaurant process. The probability generation model may also be configured to, for each segment that appears after an initial segment in the plurality of segments, receive the speaker discriminant embedding extracted from a previous adjacent segment and the speaker label assigned to the previous adjacent segment as the following feature input, the feature input being used to predict the probability that the speaker will not change for the corresponding segment. In some examples, assigning the speaker label to each segment of the speech utterance includes assigning the speaker label to each segment of the speech utterance by performing a greedy search on the probability distribution of possible speakers for the corresponding segment.

[0005] In some examples, predicting a probability distribution of possible speakers for the segment includes, when the segment occurs after an initial segment in the plurality of segments: (1) predicting a probability that a current speaker associated with the speaker label assigned to a previous adjacent segment will not change for the segment; (2) for each existing speaker associated with a corresponding speaker label previously assigned to one or more previous segments, predicting a probability that the current speaker associated with the speaker label assigned to the previous adjacent segment will change to the existing speaker; and (3) predicting a probability that the current speaker associated with the speaker label assigned to the previous adjacent segment will change to a new speaker. In some cases, the probability that the current speaker associated with the speaker label assigned to the previous adjacent segment will change to the existing speaker is proportional to the number of instances to which the corresponding speaker label associated with the existing speaker was previously assigned. Additionally or alternatively, the probability that the current speaker associated with the speaker label assigned to the previous neighboring segment will change to the new speaker is proportional to a speaker assignment probability parameter α.

[0006] In some implementations, extracting the speaker discriminant embedding from the segment includes extracting a d-vector from the segment. In other implementations, extracting the speaker discriminant embedding from the segment includes extracting an i-vector from the segment. In some configurations, the probabilistic generative model includes a recurrent neural network (RNN). In these configurations, the RNN may include a hidden layer having N gated recurrent unit (GRU) monomers; and two fully connected layers, each fully connected layer having N nodes and configured to apply rectified linear unit (ReLU) activations of the hidden layer. Each GRU monomer is configured to apply a hyperbolic tangent (tanh) activation. Additionally or alternatively, the method may also include: transcribing the speech utterance into corresponding text (152) by the data processing hardware; and annotating the text by the data processing hardware based on the speaker label assigned to each segment of the speech utterance. Segmenting the speech utterance into a plurality of segments includes: segmenting the speech utterance into a plurality of fixed-length segments or segmenting the speech utterance into a plurality of variable-length segments.

[0007] Another aspect of the present disclosure provides a system for speaker logging, the system comprising data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving a speech utterance and segmenting the speech utterance into a plurality of segments. For each segment of the speech utterance, the operations further include: extracting a speaker discriminant embedding from the segment and predicting a probability distribution of possible speakers about the segment using a probabilistic generative model configured to receive the extracted speaker discriminant embedding as a feature input. The probabilistic generative model is trained on a corpus of training speech utterances, wherein each training speech utterance is segmented into a plurality of training segments. Each training segment includes a corresponding speaker discriminant embedding and a corresponding speaker label.

[0008] This aspect may include one or more of the following optional features. In some implementations, the probability generation model predicts a probability distribution of possible speakers for each segment by applying a distance-dependent Chinese restaurant process. The probability generation model may also be configured to, for each segment that occurs after an initial segment in the plurality of segments, receive the speaker discriminant embedding extracted from a previous adjacent segment and the speaker label assigned to the previous adjacent segment as the following feature inputs, the feature inputs being used to predict the probability that the speaker will not change for the corresponding segment. In some examples, assigning the speaker label to each segment of the speech utterance includes assigning the speaker label to each segment of the speech utterance by performing a greedy search on a probability distribution of possible speakers for the corresponding segment.

[0009] In some examples, predicting a probability distribution of possible speakers for the segment includes, when the segment occurs after an initial segment in the plurality of segments: (1) predicting a probability that a current speaker associated with the speaker label assigned to a previous adjacent segment will not change for the segment; (2) for each existing speaker associated with a corresponding speaker label previously assigned to one or more previous segments, predicting a probability that the current speaker associated with the speaker label assigned to the previous adjacent segment will change to the existing speaker; and (3) predicting a probability that the current speaker associated with the speaker label assigned to the previous adjacent segment will change to a new speaker. In some cases, the probability that the current speaker associated with the speaker label assigned to the previous adjacent segment will change to the existing speaker is proportional to the number of instances to which the corresponding speaker label associated with the existing speaker was previously assigned. Additionally or alternatively, the probability that the current speaker associated with the speaker label assigned to the previous neighboring segment will change to the new speaker is proportional to a speaker assignment probability parameter α.

[0010] In some implementations, extracting the speaker discriminant embedding from the segment includes extracting a d vector from the segment. In other implementations, extracting the speaker discriminant embedding from the segment includes extracting an i vector from the segment. In some configurations, the probabilistic generative model includes a recurrent neural network (RNN). In these configurations, the RNN may include a hidden layer having N gated recurrent unit (GRU) monomers; and two fully connected layers, each fully connected layer having N nodes and configured to apply rectified linear unit (ReLU) activations of the hidden layer. Each GRU monomer is configured to apply a hyperbolic tangent (tanh) activation. Additionally or alternatively, the operation may also include transcribing the speech utterance into corresponding text and annotating the text based on the speaker label assigned to each segment of the speech utterance. Segmenting the speech utterance into multiple segments includes segmenting the speech utterance into multiple fixed-length segments or segmenting the speech utterance into multiple variable-length segments.

[0011] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A An example system for assigning speaker tags to speech utterances is schematically illustrated.

[0013] Figure 1B Schematic diagram showing the use of supervised training to train Figure 1A Example of a probabilistic generative model of a system.

[0014] Figure 2 An example process of speaker logging using a probabilistic generative model is schematically illustrated.

[0015] FIG. 3A to FIG. 3D An example recurrent neural network model configured to predict a probability distribution over possible speakers for fixed-length segments of speech utterances is schematically illustrated.

[0016] Figure 4 is a table showing the logged error rates (DERs) of the logging systems trained on the unsupervised and supervised datasets for various speaker recognition models.

[0017] Figure 5 is a flow diagram of an example arrangement of operations of a method for assigning speaker tags to speech utterances.

[0018] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0019] Like reference numbers in the various drawings represent like elements. DETAILED DESCRIPTION

[0020] Automatic speech recognition (ASR) systems typically rely on speech processing algorithms that assume that only one speaker is present in a given input audio signal. Input audio signals that include the presence of multiple speakers can potentially disrupt these speech processing algorithms, resulting in inaccurate speech recognition results output by the ASR system. As such, speaker logging is a process that segments speech from the same speaker in a larger conversation in order to determine not specifically who is speaking (speaker recognition / identification) but when someone is speaking. In other words, speaker logging includes a series of speaker recognition tasks with short utterances, and determines whether two segments of a given conversation are spoken by the same person, and is repeated for all parts of the conversation.

[0021] Existing speaker logging systems typically include multiple relatively independent components, such as but not limited to a speech segmentation module, an embedding extraction module, and a clustering module. The speech segmentation module is typically configured to remove non-speech parts from the input speech and divide the input speech into small fixed-length segments, while the embedding extraction module is configured to extract corresponding speaker discriminant embeddings from each fixed-length segment. The speaker discriminant embedding may include an i-vector or a d-vector. The clustering module used by the existing speaker logging system is responsible for determining the number of speakers present in the input speech and assigning a speaker identifier (e.g., a label) to each fixed-length segment. These clustering modules can use popular clustering algorithms, including Gaussian mixture models, mean shift clustering, agglomerative hierarchical clustering, k-means clustering, link clustering, and spectral clustering. The speaker logging system can also use additional re-segmentation modules to further refine the logging results output from the clustering module by enforcing additional constraints.

[0022] The clustering module operates in an unsupervised manner such that all speakers are assumed to be unknown, and the clustering algorithm needs to generate new "clusters" to accommodate new / unknown speakers for each new input utterance. A drawback of these unsupervised frameworks is that they cannot be improved by learning from a large set of labeled training data that includes timestamped speaker labels and ground truth. Since such labeled training data is readily available in many domain-specific applications, speaker logging systems can benefit from labeled training data by becoming more robust and more accurate in generating logged results. Furthermore, existing state-of-the-art clustering algorithms are overwhelmingly performed offline, making it difficult to generate logged results by clustering in real time.

[0023] Implementations herein are directed to a speaker logging system that implements a fully supervised probabilistic generative model for generating logging results online (e.g., in real time). The logging results include speaker labels predicted for each of a plurality of segments segmented from an input audio signal. By replacing a common clustering module that relies on an unsupervised clustering algorithm (e.g., k-means, spectral clustering, hierarchical clustering, etc.) with a fully supervised probabilistic generative model, the speaker logging system is able to improve speaker label prediction accuracy by learning from readily available timestamped speaker labels and ground truth. The segments can be fixed length or variable length.

[0024] Specifically, the probabilistic generative model includes an unbounded interleaved state recurrent neural network (UIS-RNN), which naturally incorporates timestamped speaker labels for training, so that the model models each speaker through corresponding instances that share the same parameters as other instances, generates an unbounded number of RNN instances, and interleaves the states of different RNN instances (i.e., different speakers) in the time domain. This fully supervised framework enables the model to automatically learn many speakers within each utterance via a Bayesian nonparametric process and carry information over time via the RNN.

[0025] refer to Figure 1A and Figure 1B In some implementations, the system 100 includes a user device 110 that captures speech utterances 120 from a group of speakers (e.g., users) 10, 10a-n and communicates with a remote system 140 via a network 130. The remote system 140 can be a distributed system (e.g., a cloud computing environment) with scalable / elastic resources 142. The resources 142 include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). In some implementations, the user device 110 and / or the remote system 140 executes a logging system 200 configured to receive audio data 122 corresponding to the captured utterance 120 from the user device 110, segment the audio data 122 into a plurality of segments 220, 220a-n, each segment being associated with a corresponding speaker discriminant embedding 240 extracted therefrom, and generate a logging result 280 using the probabilistic generative model 300, the logging result including a corresponding speaker label 250 assigned to each segment 220. Thus, since each segment 220 indicates a corresponding speaker discriminant embedding 240 for a corresponding time window of the input audio data 122, the logging result 280 provides the received audio data 122 with a timestamped speaker label 250 that identifies not only who is speaking during a given segment 220, but also when a speaker change occurs between adjacent segments 220. Described in more detail below, a probabilistic generative model is trained on labeled training data 202, wherein the labeled training data includes timestamped speaker labels 250T for a corpus of training speech utterances. Segments 220, 220a-n may include fixed length segments or variable length segments. Although the present disclosure may refer to one or more segments as "fixed length," the corresponding one or more segments may similarly be variable length without departing from the scope of the present disclosure. In addition, the logging system 200 may be executed on either the user device 110 or the remote system 140 without departing from the scope of the present disclosure.

[0026] In some examples, the remote system 140 also executes an automatic speech recognition (ASR) module 150 that is configured to receive the audio data 122 and transcribe it into a corresponding ASR result 152. The user device 110 can similarly execute the ASR module 150 on the device rather than on the remote system 140, which may be useful when a network connection is unavailable or when a fast (albeit lower fidelity) transcription is preferred. Additionally or alternatively, both the user device 110 and the remote system 140 can execute a corresponding ASR module 150 so that the audio data 122 can be transcribed on the device, via the remote system, or some combination thereof. In some implementations, both the ASR module 150 and the logging system 200 are executed entirely on the user device 110 and do not require any network connection to the remote system 140. The ASR result 152 may also be referred to as a "transcription" or simply as a "text." The ASR module 150 can communicate with the logging system 200 to utilize the logging results 280 associated with the audio data 122 for improving speech recognition of the audio data 122. For example, the ASR module 150 can apply different speech recognition models (e.g., language models, prosody models) for different speakers identified from the logging results 280. Additionally or alternatively, the ASR module 150 and / or the logging system 200 (or some other component) can index the transcription 152 of the audio data 122 using the predicted timestamped speaker tags 250 for each fixed-length segment 220 obtained from the logging results 280. For example, a transcription of a conversation between multiple colleagues (e.g., speakers 10) during a business meeting can be indexed by speaker to associate portions of the transcription with corresponding speakers for identifying what each speaker said.

[0027] The user device 110 includes data processing hardware 112 and memory hardware 114. The user device 110 may include an audio capture device (e.g., a microphone) for capturing speech utterances 120 from the speaker 10 and converting them into audio data 122 (e.g., electrical signals). In some implementations, the data processing hardware 112 is configured to execute a portion of the logging system 200 locally, while the remainder of the logging system 200 is executed on the remote system 140. Alternatively, the data processing hardware 112 may execute the logging system 200 instead of executing the logging system 200 on the remote system 140. The user device 110 may be any computing device capable of communicating with the remote system 140 via the network 130. The user device 110 includes, but is not limited to, desktop computing devices and mobile computing devices such as laptop computers, tablet computers, smart phones, and wearable computing devices (e.g., headphones and / or watches). The user device 110 may optionally execute an ASR module 150 to transcribe the audio data 122 into corresponding text 152. For example, when network communications are interrupted or unavailable, the user device 110 may locally execute the logging system 200 and / or the ASR module 150 to produce logging results of the audio data 122 and / or generate the transcription 152 of the audio data 122 .

[0028] In the example shown, the speaker 10 and the user device 110 may be located in an environment (e.g., a room) in which the user device 110 is configured to capture voice utterances 120 spoken by the speaker 10 and convert them into audio data 122 (also referred to as audio signals 122). For example, the speaker 10 may correspond to a colleague having a conversation during a meeting, and the user device 110 may record the voice utterances 120 and convert them into audio data 122. In turn, the user device 110 may provide the audio data 122 to the logging system 200 for assigning a timestamped speaker tag 250 to each fixed-length segment 220 of the audio data 122. Thus, the logging system 200 is responsible for processing the audio data 122 to determine when someone is speaking, without specifically determining who is speaking via speaker recognition / identification. In some examples, the user device 110 may be remote from the location of the speaker 10. For example, the user device 110 may include a remote device (e.g., a network server) that captures the voice utterances 120 from the speaker as a participant in a phone call or video conference. In this case, each speaker 10 will speak to his or her own device (e.g., phone, radio, computer, smart watch, etc.), which captures the voice utterance 120 and provides it to the remote user device 110 for conversion of the voice utterance 120 into audio data 122. Of course, in this case, the utterance 120 may undergo processing at each of the user devices and be converted into a corresponding audio signal transmitted to the remote user device 110, where the remote user device may additionally process the audio data 122 provided to the journaling system 200.

[0029] In the example shown, the logging system 200 includes a segmentation module 210, an embedding module 230, and a speaker tag predictor 260. The segmentation module 210 is configured to receive audio data 122 corresponding to a speech utterance 120 (also referred to as a "speech utterance") and segment the audio data 122 into a plurality of fixed-length segments 220. The segmentation module 210 may also remove non-speech portions from the audio data 122, for example, by applying a voice activity detector.

[0030] The embedding module 230 is configured to extract a speaker discriminative embedding 240 from each fixed length segment 220. Thereafter, the embedding module 230 provides the embedded observation sequence X=(x1, x2, . . . , x T ), where the entry x in the sequence TRepresents a real-valued speaker discriminant embedding 240 associated with a corresponding fixed-length segment 220 in the audio data 122 of the original utterance 120. The speaker discriminant embedding 240 may include speaker factors, such as d-vectors or i-vectors. Advantageously, the d-vectors may improve the logging performance due to the fact that the neural network generates the d-vectors and may be trained with large datasets, which are sufficiently robust to varying speaker accents and acoustic conditions under different usage scenarios.

[0031] In some implementations, speaker tag predictor 260 receives an embedded observation sequence X and uses probabilistic generative model 300 to generate / predict a probability distribution for each entry x in the sequence. T In other words, for each fixed length segment 220, the speaker label predictor 260 can receive the associated speaker discriminative embedding 240 extracted from the embedding module 230 as a feature input to the probability generation model 300 for generating a probability distribution of possible speakers 262 for the fixed length segment 220. The speaker label predictor 260 can assign a speaker label 250 to each fixed length segment 220 of the audio data 122 based on the probability distribution of possible speakers 262 for the fixed length segment 220. In some examples, assigning the speaker label 250 to each fixed length segment 220 includes performing a greedy search on the probability distribution of possible speakers 262 for the fixed length segment 220.

[0032] In the example shown, the speaker label predictor 260 outputs a log result 280 indicating the speaker labels 250 assigned to the corresponding fixed-length segments 220 of the audio data 122. Here, the speaker labels 250 can be represented as a speaker label sequence Y = (y1, y2, ..., y T ), where the entry y in the sequence T denotes the value assigned to the embedded entry x at time t t For example, at time t=4, speaker tag entry y4=3 corresponds to assigning the third speaker "3" to the fourth embedding entry x4. In addition, the log result 280 can predict a speaker change value 255 for each fixed length segment 220. In the example shown, the speaker change value 255 can be represented as a sequence of change point indicators Z=(z1, z2, ..., z T ), where entry z T Indicates the corresponding embedded entry x at time t t In some implementations, each change point indicator z T is a binary indicator, where z T = 1 indicates a change point at time t, while zt = 0 indicates that the speaker did not have any embedding entry x assigned to it at time t. t-1 The speaker tag 250 changes. Figure 1A In the example logging result 280 shown, when Y=(1, 1, 2, 3, 2, 2, ..., y T ), then Z = (0, 1, 1, 1, 0). Therefore, the sequence Z of change point indicators is uniquely determined by the sequence Y of speaker labels, but the sequence Y of speaker labels cannot be uniquely determined by the sequence Z of change point indicators.

[0033] refer to Figure 1B In some implementations, the probabilistic generative model 300 is trained on fully labeled training data 202 including a corpus of training speech utterances, wherein each training speech utterance is segmented into a plurality of training fixed-length segments 220T, wherein each training fixed-length segment 220T includes a corresponding speaker discriminative embedding 240T (e.g., a d-vector or an i-vector) and a corresponding speaker label 250T. Each training speech utterance in the corpus of training speech utterances can be represented as an embedded training sequence X*=(x*1, x*2, ..., x* T ), where the entry x* in the sequence T , represents a real-valued speaker discriminative embedding 240T and a corresponding speaker label 250T associated with the corresponding training fixed-length segment 220T. In addition, the fully labeled training data 202 may include a speaker change value 255T for each training fixed-length segment 220T after the initial training fixed-length segment 220T. The speaker change value 255T and the speaker label 250T may be manually assigned to each training fixed-length segment 220T. As described above, the speaker label 250T may be represented as a sequence of training speaker labels Y*=(y*1, y*2, ..., y* T ), where the entry yt in the sequence represents the speaker label 250T assigned to the embedding entry xt at time t, and the speaker change value 255T can be represented as a sequence of training change point indicators Z*=(z*1,z*2,...,z* T ), entry z t is the entry x* in the corresponding training fixed-length segment at time t t A binary indicator of whether a speaker change occurred at .

[0034] In the example shown, the plurality of training fixed-length segments 220T (each including a corresponding speaker discriminative embedding 240T (e.g., a d-vector or an i-vector) and a corresponding speaker label 250T are passed to the log trainer 204 for use in training the probabilistic generative model 300. Based on the fully labeled training data 202, the log trainer 204 can model the log parameters 206 to train the probabilistic generative model 300. Once trained, as described above with reference to Figure 1A As discussed, the probabilistic generative model (e.g., trained model) 300 is used by the speaker label predictor 260 to generate the logistic results 280 of the corresponding raw audio data 122 during inference. Therefore, in the case where the training data 202 includes known speaker labels 250T assigned to the corresponding training fixed-length segments 220T segmented from each training utterance, the trained probabilistic generative model 300 is responsible for predicting the speaker label 250 assigned to the corresponding fixed-length segment 220 segmented from the audio data 122 based solely on the corresponding speaker discriminative embedding 240 extracted from the corresponding fixed-length segment 220 segmented from the audio data 122.

[0035] The probability generation model 300 may include a neural network. The log trainer 204 maps the training data 202 to the output data to generate the neural network model 300. In general, the log trainer 204 generates hidden nodes, connection weights between hidden nodes and input nodes corresponding to the training data 202, connection weights between hidden nodes and output nodes, and connection weights between layers of hidden nodes themselves. Thereafter, the fully trained neural network model 300 may be employed for input data (e.g., raw audio data 122) to generate unknown output data (e.g., speaker labels 250).

[0036] Figure 2 A method for generating a log result 280 from the received audio data 122 is shown. Figure 1A and Figure 1B 1. Example process of the logging system 200 of FIG. 1. In the example shown, time increases from left to right, and the logging system 200 processes the audio data 122 to provide sliding windows 215 with fixed sizes and fixed overlaps. For example, the size of each window 215 can be 240 milliseconds (ms), and the fixed overlap between each sliding window 215 can be 50 percent (50%). The embedding module 230 ( Figure 1A) may include a text-independent speaker recognition network configured to extract window-level embeddings from overlapping sliding windows 215 and average the window-level embeddings to generate a segment-level d-vector, such as a speaker discriminative embedding 240. In the example shown, the text-independent speaker recognition network includes three long short-term memory (LSTM) layers and one linear layer, and is trained with a generalized end-to-end loss. In other examples, the embedding module 230 generates a segment-level i-vector. In some implementations, the segmentation module 210 ( Figure 1A ) uses a voice activity detector (VAD) with two full covariance Gaussian functions to remove non-speech portions from the audio data 122 and to segment / partition the audio data 122 into non-overlapping fixed-length segments 220. In some examples, each fixed-length segment 220 is 400 ms. Then, the speaker tag predictor 260 ( Figure 1A ) receives each fixed length segment 220 and the corresponding speaker discriminative embedding 240, and uses the probabilistic generation model 300 to generate a log result 280. Here, the log result 280 includes a speaker label 250, such as represented as a speaker label sequence Y = (y1, y2, ..., y T ), these speaker labels 250 are assigned to each fixed-length segment 220, as represented by the embedded observation sequence X = (x1, x2, ..., x T ).

[0037] FIG. 3A to FIG. 3D An example of a probabilistic generative model 300 is shown. In the example shown, the model 300 includes a recurrent neural network (RNN). While a general neural network does not store any intermediate data when processing input data to generate output data, the RNN does store data / state persistently, which can allow the RNN to improve classification capabilities relative to general neural networks that do not store data / state persistently. More specifically, the probabilistic generative model 300 includes an unbounded interleaved state recurrent neural network (UIS-RNN) trained on the training data 202 by the logging trainer 204 to model each speaker 10 in the audio data 122 of the test utterance 120 by the following corresponding instances, where the corresponding instances share the same logging parameters 206 as other instances modeling the other speakers 10. FIG. 3A to FIG. 3D , time increases from left to right. UIS-RNN300 is able to generate an unbounded number of RNN instances and interleave the states of different RNN instances (i.e., different speakers) in the time domain. tTherefore, UIS-RNN 300 maintains multiple instances, where each instance models a corresponding speaker and shares the same set of log parameters 206 with other instances. Log parameters 206 may include RNN parameters θ, speaker assignment probability parameters α, Gaussian model parameters σ 2 and the speaker changes the parameter λ.

[0038] In some examples, the UIS-RNN 300 includes a hidden layer having N gated recurrent unit (GRU) monomers and two fully connected layers, wherein the GRU monomers have hyperbolic tangent (tanh) activations, and the two fully connected layers each have N nodes and are configured to apply rectified linear unit (ReLU) activations of the hidden layer. Each GRU monomer can be composed of a corresponding hidden standard RNN state h t Represented by, and the two fully connected layers can be represented by FIG. 3A to FIG. 3D The RNN output layer shown in the figure represents that each output node m t Corresponding to the hidden standard RNN state h t and RNN parameters θ. Here, each output node m t Provide the RNN state h t to a state more suitable for generating actual observations of speaker changes. In the example shown, m t Equal to h t , but more complex functions are possible.

[0039] For a given test utterance 120 (represented by audio data 122), utterance 120 is segmented into a plurality of fixed-length segments 220 (e.g., using segmentation module 210), and speaker discriminative embeddings 240 are extracted from each fixed-length segment 220 (e.g., using embedding module 230) to provide a corresponding observation sequence X = (x1, x2, x3, x4, x5, x6, x7) of embeddings. Here, each embedding entry x in the sequence 1-7 represents the speaker discriminative embedding 240 associated with a given fixed-length segment 220. In general, each entry x in the sequence 1-7 A timestamped speaker discriminant embedding 240 corresponding to a given utterance. For example, entry x3 represents a speaker discriminant embedding 240 associated with a third fixed-length segment 220 extracted from the audio data 122 of the test utterance 120. As described in more detail below, the UIS-RNN model 300 models speaker assignment and speaker change for an unbounded number of speakers to predict a speaker label 250 to be assigned to each corresponding fixed-length segment 220, whereby the speaker label 250 is represented as a speaker label sequence Y = (y1, y2, y3, y4, y5, y6, y7).

[0040] The first embedding entry x1 (i.e., the speaker discriminative embedding 240 extracted from the first fixed-length segment 220) will always be assigned the first speaker label 250 associated with the first speaker (y1=1). For each embedding entry x2-x7 following the first entry x1 in the data sequence of the embedded observation sequence, the UIS-RNN 300 is configured to predict the speaker label y1 associated with the first speaker. t The probability distribution of the possible speakers 262 of the x-values ​​(i.e., the corresponding fixed-length segments 220 and the associated speaker discriminative embeddings 240) is calculated, and the speaker labels 250 are assigned to the corresponding entries x based on the probability distribution of the possible speakers 262. t . In some examples, speaker labels 250 are assigned by performing a greedy search over a probability distribution over possible speakers 262. The greedy search can be performed during a decoding process that implements a beam search. To model speaker assignments and speaker changes, or more specifically, speaker turning behavior, UIS-RNN 300 can use a distance-dependent Chinese restaurant process that includes a Bayesian nonparametric model configured to model an unbounded number of speakers. For example, when the next entry x in the sequence is t When modeling speaker assignments, UIS-RNN 300 predicts until the immediately preceding entry x t-1 The probability of each existing speaker assignment so far and the prediction of the next item x t The probability of the new speaker label .

[0041] At time t=1, the first speaker label associated with the first speaker y1=1 is assigned to the first embedding entry x1, and the corresponding first RNN state h1 instantiates a new RNN corresponding to the first speaker with an initial hidden state h0. Here, the first RNN state h1 has no prior knowledge of the first speaker.

[0042] At time t=2, the first speaker label y2=1 associated with the first speaker is assigned to the second embedding entry x2, and the corresponding second RNN state h2 updates the instance of the RNN corresponding to the first speaker using the previous first RNN state h1 and the previous first embedding entry x1 corresponding to the first speaker. Therefore, the updated RNN corresponding to the first speaker can be improved based on the prior knowledge obtained from the previous RNN state h1 and the previous embedding entry x1. The previous embedding entry x1 helps predict the speaker label y2.

[0043] At time t=3, a second speaker label y3=2 associated with the second speaker is assigned to the third embedding entry x3, and the corresponding third RNN state h3 instantiates a new RNN corresponding to the second speaker with the same initial hidden state h0. Since the second speaker is new (e.g., not previously seen), the third RNN state h3 has no prior knowledge of the second speaker. Furthermore, information from the first and second RNN states h1, h2 is not passed to the RNN corresponding to the second speaker or any other RNN instantiated for speakers other than the first speaker.

[0044] At time t=4, a third speaker label y4=3 associated with the third speaker is assigned to the fourth embedding entry x4, and the corresponding fourth RNN state h4 instantiates a new RNN corresponding to the third speaker with the same initial hidden state h0. Since the third speaker is new (e.g., not previously seen), the fourth RNN state h4 has no prior knowledge of the third speaker. In addition, information from the first and second RNN states h1, h2 associated with the RNN corresponding to the first speaker and the third RNN state h3 associated with the RNN corresponding to the second speaker is not passed to the RNN corresponding to the third speaker. Although the instances of the RNNs model separate speakers, the RNNs share the same set of log parameters 206 and are all initialized with the same initial hidden state h0.

[0045] At time t=5, the second speaker label y5=2 associated with the second speaker is assigned to the fifth embedding entry x5, and the corresponding fifth RNN state h5 updates the instance of the RNN corresponding to the second speaker using the previous third RNN state h3 and the previous third embedding entry x3 corresponding to the second speaker. Therefore, based on the prior knowledge obtained from the previous RNN state h3 and the previous embedding entry x3, the updated RNN corresponding to the second speaker can be improved. The previous embedding entry x3 corresponding to the second speaker helps predict the speaker label y5.

[0046] At time t=6, the second speaker label y6=2 associated with the second speaker is assigned to the sixth embedding entry x6, and the corresponding sixth RNN state h6 updates the instance of the RNN corresponding to the second speaker using the previous fifth RNN state h5 and the previous fifth embedding entry x5 corresponding to the second speaker. Therefore, the updated RNN corresponding to the second speaker can be improved based on the prior knowledge obtained from the previous RNN state h5 and the previous embedding entry x5. The previous embedding entry x5 helps predict the speaker label y6.

[0047] Therefore, in the current stage up to time t=6, FIG. 3A to FIG. 3DThe UIS-RNN 300 is shown with the predicted speaker label 250, denoted as y [6] =(1, 1, 2, 3, 2, 2) are assigned to corresponding ones of the embedded observation sequence X=(x1, x2, x3, x4, x5, x6). In the example shown, the boxes representing entries x1, x2 include horizontal lines indicating the assignment of a first speaker label associated with a first speaker, the boxes representing entries x3, x5, x6 include vertical lines indicating the assignment of a second speaker label 250 associated with a second speaker, and the box representing the fourth entry x4 includes a diagonal line indicating the assignment of a third speaker label 250 associated with a third speaker.

[0048] In the example shown, the UIS-RNN is responsible for predicting the next speaker label y7 that will be assigned to the next embedding entry x7 in the sequence. To do this, the UIS-RNN applies a distance-dependent Chinese restaurant process. In this case, there are four choices for y7: (1) the first speaker; (2) the second speaker, (3) the third speaker, or (4) the fourth speaker. Options 1-3 all include existing speakers, where the probability of each existing speaker is proportional to the number of consecutive fixed-length segments associated with the existing speaker. On the other hand, option 4 includes a probability proportional to the speaker assignment probability parameter α. Therefore, the UIS-RNN 300 predicts the next speaker label y7 based on the previous speaker label sequence y [6] and the embedded previous observation sequence x [6] Both are used to predict the probability distribution about the possible speakers, that is, the first speaker y7=1, the second speaker y7=2, the third speaker y7=3 and the fourth speaker y7=4.

[0049] refer to Figure 3A , the probability of predicting y7 to be equal to the first speaker is based on the last embedding entry x2 and the previous state h2 corresponding to the first speaker. In this case, the current RNN state h7 is modeled for the first speaker based on the second RNN state h2 and the second embedding entry x2 corresponding to the previous RNN state and previous embedding entry of the first speaker. The output of the current RNN state h7 can be interconnected with the corresponding node m7 of the RNN output, whereby m7 applies a nonlinear transformation layer of the standard RNN state h7 to generate the actual observed state (e.g., better transform the features in h7). The probability of y7=1 is proportional to the number of consecutive fixed-length segments 220 associated with the first speaker.

[0050] refer to Figure 3B, the probability that y7 is equal to the second speaker is predicted based on the last embedding entry x6 and the previous state h6 corresponding to the second speaker. In this case, no speaker change will occur, and the current RNN state h7 is modeled for the second speaker based on the sixth RNN state h6 and the sixth embedding entry x2 corresponding to the previous RNN state and previous embedding entry of the second speaker. The output of the current RNN state h7 can be interconnected with the corresponding node m7 of the RNN output, whereby m7 applies a nonlinear transformation layer of the standard RNN state h7 to generate the state actually observed (e.g., better transform the features in h7). The probability that y7=2 is proportional to the number of consecutive fixed-length segments 220 associated with the second speaker.

[0051] refer to Figure 3C , predicting the probability that y7 is equal to the third speaker is based on the last embedding entry x3 and the previous state h3 corresponding to the third speaker. In this case, the current RNN state h7 is modeled for the third speaker based on the third RNN state h3 and the third embedding entry x3 corresponding to the previous RNN state and previous embedding entry of the third speaker. The output of the current RNN state h7 can be interconnected with the corresponding node m7 of the RNN output, whereby m7 applies a nonlinear transformation layer of the standard RNN state h7 to generate the actual observed state (e.g., better transform the features in h7). The probability that y7=3 is proportional to the number of consecutive fixed-length segments 220 associated with the third speaker.

[0052] refer to Figure 3D , the probability that y7 is equal to the fourth new speaker is proportional to the speaker assignment probability parameter α. In this case, UIS-RNN 300 instantiates a new RNN for the fourth speaker, where the current RNN state h7 models the fourth speaker according to the same initial hidden state h0. Here, the new RNN instance for the fourth speaker shares the same set of logitized parameters as the RNN instances for each of the first, second, and third speakers.

[0053] In some implementations, the logging system 200 employs an online decoding method that sequentially performs a greedy search on the probability distribution (y7: 1, 2, 3, 4) of the possible speakers 262 of the corresponding fixed-length segment x7 to reduce the computational complexity from O(T!) to O(T 2 ). Based on the observation that the maximum number of speakers per utterance in most scenarios is bounded by a constant C, the computational complexity can be further reduced to O(T). In some examples, the logging system performs beam search on the decoding algorithm and adjusts multiple look-ahead entries to achieve better decoding results.

[0054] Figure 44 is a table 400 showing the logarithmic error rate (DER) for k-means clustering, spectral clustering, UIS-RNNpure 5-fold, UIS-RNN Dick-6, and UIS-RNNpure 5-fold plus additional training data for various speaker recognition models V1, V2, V3. The first speaker recognition model V1 may contain thirty-six million utterances from 18,000 US English speakers, the second speaker recognition model V2 may contain thirty-four million utterances from 138,000 non-US English speakers, and the third speaker recognition model V3 updates the second speaker recognition model V2 by using a variable length window. The results show that the largest improvement in DER comes from upgrading the speaker recognition model from V2 to V3. When using the same speaker recognition model, UIS-RNN performs significantly better than spectral offline clustering. It is also important to note that UIS-RNN predicts speaker labels in an online manner during inference. When unsupervised clustering algorithms are performed online, they typically perform significantly worse than offline clustering algorithms such as spectral clustering. Additionally, DER improves for UIS-RNN when trained on additional training data, indicating that the UIS-RNN model benefits from learning from more examples.

[0055] Figure 5 Flowchart of an example arrangement of operations of a method 500 for performing speaker logging on a received speech utterance 120. The data processing hardware 112, 144 can perform the operations of the method 500 by executing instructions stored on the memory hardware 114, 146. At operation 502, the method 500 includes segmenting the speech utterance 120 into a plurality of segments 220. The segments 220 may include fixed-length segments or variable-length segments. For each segment of the speech utterance, the method 500 includes, at operation 504, extracting a speaker discriminant embedding 240 from the segment 220, and at operation 506, using a probabilistic generative model 300 configured to receive the speaker discriminant embedding 240 as a feature input to predict a probability distribution of possible speakers 262 about the segment 220. In some examples, the speaker discriminant embedding 240 may include a d-vector. In other examples, the speaker discriminant embedding 240 includes an i-vector. At operation 508 , the method 500 includes assigning a speaker label 250 to each segment 220 of the speech utterance 120 based on the probability distribution of the possible speakers 262 for the corresponding segment 220 .

[0056] The probabilistic generative model 300 is trained on a corpus of training speech utterances, where each utterance is segmented into a plurality of training segments 220T. Each training fixed-length segment 220T includes a corresponding speaker discriminative embedding 240T and a corresponding speaker label 250T. The probabilistic generative model 300 can predict a probability distribution of possible speakers 262 for each segment 220 by applying a distance-dependent Chinese restaurant process. The probabilistic generative model 300 can include a recurrent neural network (RNN), where each speaker is modeled by a corresponding RNN instance that does not share information with other speakers' RNN instances.

[0057] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0058] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0059] Figure 6 6 is a schematic diagram of an example computing device 600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0060] The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or in other appropriate ways. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 680 coupled to the high-speed interface 640). In other implementations, multiple processors and / or multiple buses and multiple memories and memory types can be used appropriately. Moreover, multiple computing devices 600 can be connected, each of which provides a portion of the necessary operations (for example, as a server library, a blade server group, or a multi-processor system).

[0061] Memory 620 stores information non-temporarily within computing device 600. Memory 620 may be a computer-readable medium, (one or more) volatile memory units, or (one or more) non-volatile memory units. Non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0062] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device array, including a device in a storage area network or other configuration. In another implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as a memory 620, a storage device 630, or a memory on a processor 610.

[0063] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. This allocation of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0064] As shown, computing device 600 can be implemented in many different forms. For example, it can be implemented as a standard server 600a or multiple times in a group of such servers 600a, implemented as a laptop computer 600b, or implemented as part of a rack server system 600c.

[0065] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, at least one input device, and at least one output device, wherein the programmable processor may be special purpose or general purpose, coupled to receive data and instructions from a storage system and to transmit data and instructions to the storage system.

[0066] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0067] The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating outputs. These processes and logic flows can also be performed by special logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, to receive data from or transmit data to the one or more mass storage devices, or both. However, a computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0068] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball), through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0069] Many implementations have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A method (500), comprising: Receiving a speech utterance (120) at data processing hardware (200); Segmenting the speech utterance (120) into a plurality of segments (220) by the data processing hardware (200); For each segment (220) of the speech utterance (120): extracting, by the data processing hardware (200), speaker discriminative embeddings (240) from the segments (220); as well as predicting, by the data processing hardware (200), a probability distribution of possible speakers (262) of the segment (220) using a probabilistic generative model (300), the probabilistic generative model (300) being configured to receive the extracted speaker discriminative embedding (240) as a feature input, the probabilistic generative model (300) being trained on a corpus of training speech utterances (120), each training speech utterance (120) being segmented into a plurality of training segments (220T), each training segment (220T) comprising a corresponding speaker discriminative embedding (240) and a corresponding speaker label (250); assigning (220), by the data processing hardware (200), a speaker label (250) to each segment (220) of the speech utterance (120) based on the probability distribution of possible speakers (262) corresponding to the segment; identifying, by the data processing hardware (200), different possible speakers (262) based on the assigned speaker tags (250); and Different speech recognition models are applied by the data processing hardware (200) for the different possible speakers (262).

2. The method (500) according to claim 1, wherein: The probability generation model (300) predicts the probability distribution of possible speakers (262) for each segment (220) by applying the distance-dependent Chinese restaurant process.

3. The method (500) of claim 1, wherein: Predicting a probability distribution of possible speakers (262) of the segment (220) includes, when the segment (220) occurs after an initial segment (220) in the plurality of segments (220): predicting a probability that a current speaker (10) associated with the speaker label (250) assigned to a previous neighboring segment (220) will not change for the segment (220); For each existing speaker (10) associated with a corresponding speaker label (250) previously assigned to one or more previous segments (220), predicting a probability that the current speaker (10) associated with the speaker label (250) assigned to the previous adjacent segment (220) will change to the existing speaker (10); and A probability is predicted that the current speaker (10) associated with the speaker label (250) assigned to the previous neighboring segment (220) will change to a new speaker (10).

4. The method (500) according to claim 3, wherein: The probability that the current speaker (10) associated with the speaker label (250) assigned to the previous neighboring segment (220) will change to the existing speaker (10) is proportional to the number of instances that were previously assigned the corresponding speaker label (250) associated with the existing speaker (10).

5. The method (500) of claim 3, wherein: The probability that the current speaker (10) associated with the speaker label (250) assigned to the previous neighboring segment (220) will change to the new speaker (10) is proportional to a speaker (10) assignment probability parameter a.

6. The method (500) of claim 1, wherein: The probability generation model (300) is also configured to receive, for each segment (220) after an initial segment (220) that appears in the plurality of segments (220), the speaker discriminant embedding (240) extracted from a previous adjacent segment (220) and the speaker label (250) assigned to the previous adjacent segment (220) as feature inputs, the feature inputs being used to predict the probability that the speaker (10) will not change for the corresponding segment (220).

7. The method (500) of claim 1, wherein: Assigning the speaker label (250) to each segment (220) of the speech utterance (120) includes assigning the speaker label (250) to each segment (220) of the speech utterance (120) by performing a greedy search over a probability distribution of possible speakers (10) for the corresponding segment (220).

8. The method (500) of claim 1, wherein: Extracting the speaker discriminative embedding (240) from the segment (220) includes extracting a d-vector from the segment (220).

9. The method (500) of claim 1, wherein: Extracting the speaker discriminative embedding (240) from the segment (220) includes extracting an i-vector from the segment (220).

10. The method (500) of claim 1, wherein: The probability generation model (300) includes a recurrent neural network (RNN).

11. The method (500) of claim 10, wherein: The RNN includes: a hidden layer having N gated recurrent unit (GRU) cells, each GRU cell being configured to apply a hyperbolic tangent (tanh) activation; and Two fully connected layers, each fully connected layer has N nodes and is configured to apply a rectified linear unit (ReLU) activation of the hidden layer.

12. The method (500) of claim 1, further comprising: transcribing the speech utterance (120) into corresponding text (152) by the data processing hardware (200); as well as The text is annotated by the data processing hardware (200) based on the speaker tags (250) assigned to each segment (220) of the speech utterance (120).

13. The method (500) according to any one of claims 1 to 12, wherein: Segmenting the speech utterance (120) into a plurality of segments (220) includes segmenting the speech utterance (120) into a plurality of fixed length segments (220).

14. The method (500) according to any one of claims 1 to 12, wherein: Segmenting the speech utterance (120) into a plurality of segments (220) includes segmenting the speech utterance (120) into a plurality of variable length segments (220).

15. A system (100), comprising: Data processing hardware (200); as well as Memory hardware (114, 146) in communication with the data processing hardware (200) and storing instructions that, when executed by the data processing hardware (200), cause the data processing hardware (200) to perform operations comprising: Receiving a voice utterance (120); Segmenting the speech utterance (120) into a plurality of segments (220); For each segment (220) of the speech utterance (120): extracting speaker discriminative embeddings (240) from the segments (220); and predicting a probability distribution of possible speakers (262) of the segment (220) using a probabilistic generative model (300), the probabilistic generative model (300) being configured to receive the extracted speaker discriminative embedding (240) as a feature input, the probabilistic generative model (300) being trained on a corpus of training speech utterances (120), each training speech utterance (120) being segmented into a plurality of training segments (220T), each training segment (220T) comprising a corresponding speaker discriminative embedding (240) and a corresponding speaker label (250); assigning a speaker label (250) to each segment (220) of the speech utterance (120) based on the probability distribution of possible speakers (262) for the corresponding segment; identifying different possible speakers (262) based on the assigned speaker tags (250); and Different speech recognition models are applied to the different possible speakers (262).

16. The system (100) of claim 15, wherein: The probability generation model (300) predicts the probability distribution of possible speakers (262) for each segment (220) by applying the distance-dependent Chinese restaurant process.

17. The system (100) of claim 15, wherein: Predicting a probability distribution of possible speakers (262) of the segment (220) includes, when the segment (220) occurs after an initial segment (220) in the plurality of segments (220): predicting a probability that a current speaker (10) associated with the speaker label (250) assigned to a previous neighboring segment (220) will not change for the segment (220); For each existing speaker (10) associated with a corresponding speaker label (250) previously assigned to one or more previous segments (220), predicting a probability that the current speaker (10) associated with the speaker label (250) assigned to the previous adjacent segment (220) will change to the existing speaker (10); and A probability is predicted that the current speaker (10) associated with the speaker label (250) assigned to the previous neighboring segment (220) will change to a new speaker (10).

18. The system (100) of claim 17, wherein: The probability that the current speaker (10) associated with the speaker label (250) assigned to the previous neighboring segment (220) will change to the existing speaker (10) is proportional to the number of instances that were previously assigned the corresponding speaker label (250) associated with the existing speaker (10).

19. The system (100) of claim 17, wherein: The probability that the current speaker (10) associated with the speaker label (250) assigned to the previous neighboring segment (220) will change to the new speaker (10) is proportional to a speaker (10) assignment probability parameter a.

20. The system (100) of claim 15, wherein: The probability generation model (300) is also configured to receive, for each segment (220) after an initial segment (220) that appears in the plurality of segments (220), the speaker discriminant embedding (240) extracted from a previous adjacent segment (220) and the speaker label (250) assigned to the previous adjacent segment (220) as feature inputs, the feature inputs being used to predict the probability that the speaker (10) will not change for the corresponding segment (220).

21. The system (100) of claim 15, wherein: Assigning the speaker label (250) to each segment (220) of the speech utterance (120) includes assigning the speaker label (250) to each segment (220) of the speech utterance (120) by performing a greedy search over a probability distribution of possible speakers (10) for the corresponding segment (220).

22. The system (100) of claim 15, wherein: Extracting the speaker discriminative embedding (240) from the segment (220) includes extracting a d-vector from the segment (220).

23. The system (100) of claim 15, wherein: Extracting the speaker discriminative embedding (240) from the segment (220) includes extracting an i-vector from the segment (220).

24. The system (100) of claim 15, wherein: The probability generation model (300) includes a recurrent neural network (RNN).

25. The system (100) of claim 24, wherein: The RNN includes: a hidden layer having N gated recurrent unit (GRU) cells, each GRU cell being configured to apply a hyperbolic tangent (tanh) activation; and Two fully connected layers, each fully connected layer has N nodes and is configured to apply a rectified linear unit (ReLU) activation of the hidden layer.

26. The system (100) of claim 15, wherein: The operations also include: transcribing the speech utterance (120) into corresponding text (152); and The text is annotated based on the speaker tags (250) assigned to each segment (220) of the speech utterance (120).

27. The system (100) according to any one of claims 15-26, wherein: Segmenting the speech utterance (120) into a plurality of segments (220) includes segmenting the speech utterance (120) into a plurality of fixed length segments (220).

28. The system (100) according to any one of claims 15-26, wherein: Segmenting the speech utterance (120) into a plurality of segments (220) includes segmenting the speech utterance (120) into a plurality of variable length segments (220).

Images