Evaluation metrics for rating-based speaker change detection

Abstract

The method (600) includes obtaining a multi-utterance training sample (410) including audio data (412) characterizing utterances spoken by two or more different speakers (10) and obtaining ground truth speaker change intervals (414) indicating time intervals in the audio data during which speaker change occurs between the two or more different speakers. The method also includes processing the audio data using a sequence transfer model (300) to generate a sequence of predicted speaker change tokens (302). The method includes, for each corresponding predicted speaker change token, labeling the corresponding predicted speaker change token as a correct answer when the predicted speaker change token overlaps with one of the ground truth speaker change intervals. The method also includes determining an accuracy metric (442) for the sequence transfer model based on the number of predicted speaker change tokens labeled as correct and the total number of predicted speaker change tokens.

Description

[Technical Field]

[0001] TECHNICAL FIELD This disclosure relates to evaluation metrics for evaluation-based speaker change detection. [Background technology]

[0002] Speaker change detection is the process of receiving input audio data and outputting speaker change tokens that identify speaker transition points (e.g., when one speaker stops speaking and another speaker begins speaking) in a multi-speaker conversation. Traditionally, speaker change detection maps input acoustic features to a frame-level binary prediction that indicates whether a speaker change occurred. However, training a model to perform speaker change detection can be affected by the large variability present in most annotated training data. That is, identifying the exact time when a speaker transition point occurred is highly subjective and dependent on the person annotating the training data. Therefore, the large variability in the annotated training data can result in poor performance of a model performing speaker change detection. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for evaluating speaker change detection in a continuous, conversational input audio stream with multiple speakers. The operations include acquiring a plurality of utterance samples including audio data characterizing utterances spoken by two or more different speakers. The operations also include acquiring ground truth speaker change intervals indicating time intervals in the audio data during which speaker changes between the two or more different speakers occur. The operations also include processing the audio data using a sequence transfer model to generate a sequence of predicted speaker change tokens, each indicating the location of a respective speaker change within the audio data. For each corresponding predicted speaker change token, the operations include labeling the corresponding predicted speaker change token as a correct answer when the predicted speaker change token overlaps with one of the ground truth speaker change intervals. The operations also include determining an accuracy metric for the sequence transfer model based on the number of predicted speaker change tokens labeled as correct and the total number of predicted speaker change tokens in the sequence of predicted speaker change tokens.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, determining an accuracy metric for the sequence conversion model is based on a ratio of the number of predicted speaker change tokens labeled as correct and the total number of predicted speaker change tokens in the sequence of predicted speaker change tokens; For each corresponding predicted speaker change token, the operations may further include labeling the corresponding predicted speaker change token as a false-accept prediction when the corresponding speaker change token does not overlap with any of the ground truth speaker change intervals.

[0005] In some examples, the operations further include, for each ground truth speaker change interval, labeling the corresponding ground truth speaker change interval as a correct match when any of the predicted speaker change tokens overlap with the corresponding ground truth speaker change interval, and determining a recall metric for the sequence transfer model based on the duration of the ground truth speaker change interval labeled as a correct match and the total duration of all the ground truth speaker change intervals. In these examples, the operations may further include determining a performance score for the sequence transfer model based on the precision metric and the recall metric. Here, determining the performance score includes calculating the performance score based on the formula: 2 × (precision metric × recall score) / (precision metric + recall score).

[0006] In some implementations, the training samples of multiple utterances further include ground truth speaker labels paired with the audio data, each ground truth speaker label indicating a corresponding time-stamped segment in the audio data associated with each utterance spoken by one of two or more different speakers, and obtaining the ground truth speaker change intervals includes identifying, as a corresponding ground truth speaker change interval, each time interval in which two or more of the time-stamped segments overlap, and identifying each time gap indicating a pause between two adjacent time-stamped segments in the audio data associated with each of the utterances spoken by the two different speakers. In these implementations, the operations may further include determining a minimum start time and a maximum start time for the audio data based on the time-stamped segments indicated by the ground truth speaker labels, and excluding predicted speaker change tokens having a timestamp earlier than the minimum start time or later than the maximum start time from determining the accuracy metric for the sequence-to-sequence model. In some examples, determining the accuracy metric for the sequence-to-sequence model is not based on any word-level speech recognition results output by the sequence-to-sequence model. Determining an accuracy metric may not require performing a full speaker diarization on the audio data.

[0007] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a plurality of speech samples including audio data characterizing speech spoken by two or more different speakers. The operations also include obtaining ground truth speaker change intervals indicating time intervals in the audio data during which speaker changes between the two or more different speakers occur. The operations also include processing the audio data using a sequence transfer model to generate a sequence of predicted speaker change tokens, each indicating a position of each speaker change within the audio data. For each corresponding predicted speaker change token, the operations include labeling the corresponding predicted speaker change token as a correct answer when the predicted speaker change token overlaps with one of the ground truth speaker change intervals. The operations also include determining an accuracy metric for the sequence transfer model based on the number of predicted speaker change tokens labeled as correct and the total number of predicted speaker change tokens in the sequence of predicted speaker change tokens.

[0008] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, determining an accuracy metric for the sequence conversion model is based on a ratio of the number of predicted speaker change tokens labeled as correct and the total number of predicted speaker change tokens in the sequence of predicted speaker change tokens; For each corresponding predicted speaker change token, the operations may further include labeling the corresponding predicted speaker change token as a false-accept prediction when the corresponding speaker change token does not overlap with any of the ground truth speaker change intervals.

[0009] In some examples, the operations further include, for each ground truth speaker change interval, labeling the corresponding ground truth speaker change interval as a correct match when any of the predicted speaker change tokens overlap with the corresponding ground truth speaker change interval, and determining a recall metric for the sequence transfer model based on the duration of the ground truth speaker change interval labeled as a correct match and the total duration of all the ground truth speaker change intervals. In these examples, the operations may further include determining a performance score for the sequence transfer model based on the precision metric and the recall metric. Here, determining the performance score includes calculating the performance score based on the formula: 2 × (precision metric × recall score) / (precision metric + recall score).

[0010] In some implementations, the training samples of multiple utterances further include ground truth speaker labels paired with the audio data, each ground truth speaker label indicating a corresponding time-stamped segment in the audio data associated with each utterance spoken by one of two or more different speakers, and obtaining the ground truth speaker change intervals includes identifying, as a corresponding ground truth speaker change interval, each time interval in which two or more of the time-stamped segments overlap, and identifying each time gap indicating a pause between two adjacent time-stamped segments in the audio data associated with each of the utterances spoken by the two different speakers. In these implementations, the operations may further include determining a minimum start time and a maximum start time for the audio data based on the time-stamped segments indicated by the ground truth speaker labels, and excluding predicted speaker change tokens having a timestamp earlier than the minimum start time or later than the maximum start time from determining the accuracy metric for the sequence-to-sequence model. In some examples, determining the accuracy metric for the sequence-to-sequence model is not based on any word-level speech recognition results output by the sequence-to-sequence model. Determining an accuracy metric may not require performing a full speaker diarization on the audio data.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an exemplary automatic speech recognition system including an automatic speech recognition model and a sequence-to-sequence model. [Figure 2] FIG. 1 is a schematic diagram of an exemplary automatic speech recognition model. [Figure 3] FIG. 1 is a schematic diagram of an exemplary sequence transformation model. [Figure 4] FIG. 1 is a schematic diagram of an exemplary two-stage training process for training a sequence transformation model. [Figure 5] 1 is a graphical representation of several components for calculating precision and recall metrics. [Figure 6] 1 is a flowchart of an exemplary arrangement of operations for a computer-implemented method of evaluation metrics for evaluation-based speaker change detection. [Figure 7] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0013] Like reference symbols in the various drawings indicate like elements.

[0014] In addition to converting input sequences into output sequences, sequence-transformation models have also been constructed to detect special input conditions and generate special outputs (e.g., special output tokens or other types of indications) when the special input conditions are detected. That is, a sequence-transformation model may be constructed and trained to process a sequence of input data and generate predicted output sequences that include, in addition to other normal / general predicted outputs (e.g., graphemes, word fragments, and / or words), a special output when the sequence-transformation model detects a corresponding special input condition in the input data. For example, a sequence-transformation model may process input audio features and generate a transcription representing the input audio features and a speaker-change token "" indicating each speaker change in a multi-speaker conversation. <st>". A turn-taking, as used herein, refers to a point in a conversation where one speaker stops speaking and / or another speaker begins speaking. Thus, a speaker change token indicates a point in a conversation where a turn-taking occurs.

[0015] However, training conventional sequence-to-sequence models has several limitations. For example, conventional systems require precise timing information of speaker change points in the training data, which is difficult for human annotators because determining where to mark speaker change points is a highly subjective process. Furthermore, methods that use purely acoustic information ignore the rich semantic information in the audio signal to identify speaker change points.

[0016] Accordingly, embodiments herein are directed to methods and systems for performing a rating-based speaker change detection and evaluation method. In particular, a training process trains a sequence-transfer model by obtaining training samples of multiple utterances, including audio data characterizing utterances spoken by two or more different speakers, and obtaining ground truth speaker change intervals indicating time intervals in the audio data during which speaker changes occur. The sequence-transfer model processes the audio data to generate a sequence of predicted speaker change tokens. The training process then labels each predicted speaker change token in the sequence of predicted speaker change tokens as a correct answer when the predicted speaker change token overlaps with one of the ground truth speaker change intervals. The training process determines an accuracy metric for the sequence-transfer model based on the number of predicted speaker change tokens labeled as correct and the total number of predicted speaker change tokens in the sequence of predicted speaker change tokens, and uses the accuracy metric to train the sequence-transfer model. As described below, the training process can train the sequence-transfer model using a recall metric in addition to or instead of the accuracy metric.

[0017] 1 , an exemplary system 100 includes a user device 110 that captures audio utterances 106 spoken by two or more speakers 10, 10a-n during a conversation and communicates with a remote system 140 via a network 130. The remote system 140 may be a distributed system (e.g., a cloud computing environment) having scalable / elastic resources 142. The resources 142 include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware). The user device 110 includes the data processing hardware 112 and the memory hardware 114. The user device 110 includes desktop and mobile computing devices, such as, but not limited to, laptops, tablets, smartphones, smart speakers / displays, smart appliances, Internet of Things (IoT) devices, and wearable computing devices (e.g., headsets and / or watches). The user device 110 may include an audio capture device (e.g., a microphone) for capturing and converting voice utterances 106 from two or more speakers 10 into an input audio stream (e.g., a sequence of audio data or acoustic frames) 108.

[0018] User device 110 and / or cloud computing environment 140 may execute an automatic speech recognition (ASR) system 118. In some implementations, user device 110 is configured to execute a portion of ASR system 118 locally (e.g., using data processing hardware 112), with the remainder of ASR system 118 executing in cloud computing environment 140 (e.g., using data processing hardware 144). Alternatively, ASR system 118 can execute entirely in user device 110 or cloud computing environment 140.

[0019] The ASR system 118 is configured to receive an input audio stream 108 corresponding to a continuous conversation between multiple speakers and to generate a speaker change detection output 125. More specifically, the ASR system 118 includes an ASR model 200 configured to receive the input audio stream 108 and generate, as an output, a transcription (e.g., a speech recognition result / hypothesis) 120 based on the input audio stream 108. Additionally, the ASR system 118 includes a sequence-to-sequence model (i.e., a speaker change detection model) 300 configured to receive the input audio stream 108 and generate, as an output, a sequence of predicted speaker change tokens 302, each indicating the location of each speaker change in the input audio stream 108. The ASR system 118 generates the speaker change detection output 125, which includes the transcription 120 generated by the ASR model 200 and the sequence of predicted speaker change tokens 302 generated by the sequence-to-sequence model 300. In some examples, the speaker change detection output 125 includes a sequence of timestamps 122 corresponding to the transcription 120 and a sequence of predicted speaker change tokens 302. Thus, in these examples, the ASR system 118 can align the transcription 120 and the sequence of predicted speaker change tokens 302 based on the corresponding sequence of timestamps 122.

[0020] For example, in the illustrated example, a first speaker 10a speaks a first utterance 106a of "Hey, how are you doing?" and a second speaker 10b speaks a second utterance 106b of "Yeah, I'm doing great." The ASR system 118 receives an input audio stream 108 corresponding to a continuous multi-speaker conversational input (e.g., the first utterance 106a and the second utterance 106b) spoken by the first and second speakers 10a, 10b. In the illustrated example, the ASR model 200 generates a transcription 120 of "Hey, how are you doing? Yeah, I'm doing great," and the sequence conversion model 300 generates a sequence of predicted speaker change tokens 302 indicating speaker changes at a fifth timestamp 122 (e.g., T=5) and a ninth timestamp 122 (e.g., T=9). In particular, the predicted speaker change token 302 at the fifth timestamp indicates the transition point where the first speaker 10a stops speaking and the second speaker 10b starts speaking, and the predicted speaker change token 302 at the ninth timestamp indicates the transition point where the second speaker 10b stops speaking.

[0021] In some implementations, two or more speakers 10 and user devices 110 may be disposed in an environment (e.g., a room) in which the user devices 110 are configured to capture and convert speech utterances 106 spoken by the two or more speakers 10 into an input audio stream 108. For example, the two or more speakers 10 may be colleagues conversing during a meeting, and the user devices 110 may record and convert the speech utterances 106 into the input audio stream 108. The user devices 110 may then provide the input audio stream 108 to an ASR system 118 to generate a speech recognition result 120 and a speaker change detection output 125 that includes a sequence of predicted speaker change tokens 302.

[0022] In some examples, at least some of the speech utterances 106 conveyed in the input audio stream 108 overlap, such that at least two speakers 10 are speaking simultaneously at a given moment. Notably, when the sequence of acoustic frames 108 is provided as input to the ASR system 118, the number N of the two or more speakers 10 may be unknown, in which case the ASR system 118 estimates the number N of the two or more speakers 10. In some implementations, the user device 110 is located remotely from one or more of the two or more speakers 10. For example, the user device 110 may include a remote device (e.g., a network server) that captures speech utterances 106 from two or more speakers 10 who are participants in a teleconference or videoconference. In this scenario, each speaker 10 speaks into a respective user device 110, which captures the speech utterances 106 and provides the speech utterances 106 to the remote user device for conversion into the input audio stream 108. Of course, in this scenario, the voice utterances 106 may be processed at each user device 110, converted into a corresponding input audio stream 108, and then transmitted to the remote user device, where the input audio stream 108 may be provided as input to the ASR system 118 for further processing.

[0023]

number

[0024]

number

[0025] The softmax layer 240 may use any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this way, the RNN-T model 200 does not make a conditional independence assumption; rather, the prediction of each symbol depends not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 assumes that the output symbols are independent of future acoustic frames 108, which allows the RNN-T model to be used in a streaming manner, a non-streaming manner, or some combination thereof.

[0026] In some examples, the audio encoder 210 of the RNN-T model includes multiple multi-head (e.g., eight-head) self-attention layers. For example, the multiple multi-head self-attention layers may include conformer layers (e.g., conformer encoders), transformer layers, performer layers, convolutional layers (including lightweight convolutional layers), or any other type of multi-head self-attention layer. The multiple multi-head self-attention layers may include any number of layers, for example, 16 layers. Furthermore, the audio encoder 210 may operate in a streaming manner (e.g., the audio encoder 210 outputs initial higher-level feature representations as soon as they are generated), a non-streaming manner (e.g., the audio encoder 210 outputs subsequent higher-level feature representations by processing additional right-hand context to improve the initial higher-level feature representation), or a combination of both streaming and non-streaming manners.

[0027]

number

[0028]

number

[0029] The probability distributions may then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, word fragments, and / or words) in a beam search process (e.g., by a softmax layer 340) that determines a sequence output 345. As described in more detail with reference to FIG. 4, the sequence output 345 generated by the sequence conversion model 300 includes a transcription 342 (or a probability distribution over possible speech recognition hypotheses) and a sequence of predicted speaker change tokens 302 during training. In particular, each possible speech recognition hypothesis is associated with a respective sequence of predicted speaker change tokens 302. In some examples, each transcription 342 may have a sequence of predicted speaker change tokens 302 embedded directly within the transcription 342, e.g., "Hey, how are you?" <st>Yeah, I'm doing fine <st>On the other hand, as shown in FIG. 1, the sequence transformation model 300 outputs only a sequence of predicted speaker change tokens 302, and does not output a transcription 342.

[0030] 4 shows an exemplary two-stage training process 400 for training a sequence transformation model 300. The exemplary two-stage training process 400 (also referred to simply as "training process 400") may be executed on a user device 110 and / or a cloud computing environment 140. The training process 400 obtains a plurality of multi-utterance training samples 410 for training the sequence transformation model 300. As described below, the two-stage training process 400, in a first stage, trains the sequence transformation model 300 by determining a negative log-likelihood loss term 422 for each multi-utterance training sample 410 of the plurality of multi-utterance training samples 410 and updates parameters of the sequence transformation model 300 based on the negative log-likelihood loss term 422. The two-stage training process 400 then fine-tunes the sequence transfer model 300 trained during the first stage in a second stage by determining a precision loss 442 and a recall loss 444 for each multi-utterance training sample 410 among the plurality of multi-utterance training samples 410, and updating the parameters of the sequence transfer model 300 based on the precision loss 442 and the recall loss 444.

[0031] Each multi-utterance training sample 410 includes audio data 412 characterizing utterances spoken by two or more different speakers. The audio data 412 of each multi-utterance training sample 410 may be paired with a corresponding ground truth speaker change interval 414 indicating a time interval in the audio data 412 during which a speaker change between the two or more different speakers occurs. For example, in a conversation in which speaker A spoke from 0.1 to 10.5 seconds and speaker B spoke from 10.8 to 15.3 seconds, the ground truth speaker change interval 414 indicates the time interval from 10.5 to 10.8 seconds during which the speaker change occurred. Thus, in this example, the training process 400 labels any predicted speaker change between the time intervals of 10.5 and 10.8 as correct. Additionally, the training process 400 may apply a collar (e.g., a 250 millisecond collar) so that the time interval is 10.75 to 11.05 seconds instead of 10.5 to 10.8 seconds in the above example. Further, the audio data 412 of each multi-utterance training sample 410 may be paired with a ground truth transcription 416 that indicates a textual representation of what was spoken in the audio data 412. That is, as discussed above, the sequence transformation model 300 may optionally output a transcription 342 during the training process 400.

[0032] In some implementations, each multi-utterance training sample 410 includes a ground truth speaker label 418 paired with audio data 412, whereby each ground truth speaker label 418 indicates a corresponding time-stamped segment in the audio data 412 associated with a respective one (or more) of the utterances spoken by one of two or more different speakers. In these implementations, the training process 400 obtains the ground truth speaker change intervals 414 by identifying each time interval in which two or more of the time-stamped segments overlap as a corresponding ground truth speaker change interval 414, and identifying each time gap indicating a pause between two adjacent time-sampled segments in the audio data associated with each of the utterances spoken by the two different speakers as a corresponding ground truth speaker change interval.

[0033] 5 shows a graphical representation 500 of various components of the training process 400 (FIG. 4), with the x-axis showing timestamps (e.g., Tmin-Tmax) increasing from left to right. The graphical representation 500 includes a first ground truth speaker label 418, 418a indicating when speaker A was speaking, a second ground truth speaker label 418, 418b indicating when speaker B was speaking, and a third ground truth speaker label 418, 418c indicating when speaker C was speaking. Furthermore, the training process 400 (FIG. 4) may obtain ground truth speaker change intervals 414 by identifying where two or more ground truth speaker labels 418 overlap as corresponding ground truth speaker change intervals 414. For example, in the illustrated example, the first ground truth speaker label 418a and the third ground truth speaker label 418c overlap between timestamps T2 and T3, causing the training process 400 to identify the time interval between timestamps T2 and T3 as the corresponding ground truth speaker change interval 414. In some implementations, the training process 400 obtains the ground truth speaker change interval 414 by identifying where there is a time gap indicative of a pause between two adjacent ground truth speaker labels 418. In the illustrated example, there is a time gap indicative of a pause between the first ground truth speaker label 418a and the second ground truth speaker label 418b between timestamps T4 and T5, causing the training process 400 to identify the time interval between timestamps T4 and T5 as the corresponding ground truth speaker change interval 414.

[0034] In some implementations, the training process 400 (FIG. 4) determines a minimum start time (Tmin) and a maximum start time (Tmax) for the audio data 412 based on time-stamped segments indicated by the ground truth speaker labels 418. In these implementations, the training process 400 excludes predicted speaker change tokens 302 having timestamps earlier than the minimum start time or later than the maximum start time when determining the accuracy metric 442 (FIG. 4) for the sequence conversion model 300. For example, in the example shown in FIG. 5, the training process 400 excludes predicted speaker change tokens 302 to the left of Tmin or to the right of Tmax.

[0035] 4, during the first stage of the training process 400, the sequence transformation model 300 processes the audio data 412 of each multi-utterance training sample 410 to generate a corresponding first-stage output sequence 345, 345a, including a respective first-stage transcription 342, 342a and a respective first-stage sequence of predicted speaker change tokens 302, 302a. The sequence transformation model 300 can determine whether to output a predicted speaker change token 302 based on corresponding semantic information of the transcription 342. The log-likelihood loss function module 420 then determines a negative log-likelihood loss term 422 based on comparing the corresponding first-stage output sequence 345a to the associated ground truth speaker change interval 414 and associated ground truth transcription 416. That is, the log-likelihood loss function module 420 compares each first-stage transcription 342a with the corresponding ground truth transcription 416 and compares each first-stage sequence of predicted speaker change tokens 302a with the corresponding ground truth speaker change intervals 414 to determine a negative log-likelihood loss term 422. More specifically, the log-likelihood loss function module 420 determines whether the first-stage predicted speaker change tokens 302a overlap with any of the ground truth speaker change intervals 414, where overlap between the first-stage predicted speaker change tokens 302a and the ground truth speaker change intervals 414 indicates a correct prediction by the sequence transfer model 300. The first stage of the training process 400 trains the sequence transfer model 300 by updating its parameters based on the negative log-likelihood loss term 422 determined for each multi-utterance training sample 410.

[0036] Then, during the second stage, the training process 400 warm-starts the sequence-transfer model 300 trained during the first stage of the training process 400. After warm-starting the sequence-transfer model 300, the second stage of the training process 400 fine-tunes the model using the same multiple-utterance training samples 410 used to train the sequence-transfer model 300 during the first stage. Notably, during the second stage of the training process 400, the sequence-transfer model 300 uses the updated parameters obtained from the first stage of the training process 400. That is, the sequence-transfer model 300 processes the audio data 412 of each multiple-utterance training sample 410 to generate a respective second-stage transcription 342, 342b and a corresponding second-stage sequence output 345, 345b, including a respective second-stage sequence of predicted speaker-change tokens 302, 302b.

[0037] The labeler 430 receives the second-stage sequence output 345b generated by the sequence-transformation model 300 for each multi-utterance training sample 410 and performs a beam search to select both the N-best transcriptions from the second-stage transcriptions 342b and the second-stage sequences of predicted speaker-change tokens 302b corresponding to the selected N-best transcriptions. For example, the second-stage transcriptions 342b may include 10 candidate transcriptions based on the corresponding multi-utterance training sample 410, whereby the labeler 430 selects the associated top three candidate transcriptions with the highest confidence scores and the corresponding second-stage sequences of predicted speaker-change tokens 302b. For each second-stage predicted speaker-changed token 302 (e.g., selected by the labeler 430), the labeler 430 generates a token label 432 that labels the corresponding second-stage predicted speaker-changed token 302b as either correct, incorrectly accepted, or incorrectly rejected.

[0038] The correct token label 432 indicates that the second-stage predicted speaker-change token 302b output by the sequence transfer model 300 correctly predicted a speaker change in the audio data 412. That is, when each second-stage predicted speaker-change token 302b overlaps with one of the ground truth speaker change intervals 414, the labeler 430 generates a token label 432 that labels each second-stage predicted speaker-change token 302b as correct. For example, as shown in FIG. 5 , the sequence transfer model 300 outputs each second-stage predicted speaker-change token 302b that indicates a speaker change that occurred between timestamps T8 and T9, which overlaps with the ground truth speaker change interval 414 that occurs between timestamps T8 and T9. Thus, the token label 432 between timestamps T8 and T9 indicates that each second-stage predicted speaker-change token 302b is correct (indicated by an “X”).

[0039] In some examples, the false-accept token label 432 indicates that the sequence transfer model 300 output a respective second-stage predicted speaker-change token 302b, but that no speaker change actually occurred in the audio data 412 (e.g., as indicated by the ground truth speaker change interval 414). Here, the labeler 430 labels a corresponding second-stage predicted speaker-change token 302b as a false-accept prediction when the corresponding second-stage predicted speaker-change token 302b does not overlap with any of the ground truth speaker change intervals 414. As shown in FIG. 5, the sequence transfer model 300 outputs each second-stage predicted speaker-change token 302b that does not overlap with any of the ground truth speaker change intervals 414, indicating that a speaker change occurred between timestamps T1 and T2. Thus, the token label 432 between timestamps T1 and T2 indicates that the respective second-stage predicted speaker-change token 302b is a false-accept prediction (denoted by “FA”).

[0040] In yet another example, the false-rejection token label 432 indicates that the sequence transfer model 300 did not output a respective second-stage predicted speaker change token 302b, but that a speaker change did in fact occur in the audio data 412 (e.g., as indicated by the ground truth speaker change interval 414). Here, the labeler 430 generates the false-rejection token label 432 when a ground truth speaker change interval 414 occurred but no second-stage predicted speaker change token 302b overlapping the ground truth speaker change interval 414 was output. As shown in FIG. 5 , although a respective ground truth speaker change interval 414 exists between timestamp T4 and timestamp T5, the sequence transfer model 300 did not output any second-stage predicted speaker change token 302b between timestamps T4 and T5. Therefore, the token label 432 between times T4 and T5 indicates a false-rejection prediction (denoted by “FR”).

[0041] The token-level loss function 440 receives the token labels 432 generated by the labeler 430 for each multi-utterance training sample 410 and determines a precision metric 442 and a recall metric 444. That is, the token-level loss function module 440 determines the precision metric 442 based on the number of second-stage predicted speaker-changed tokens 302b labeled as correct and the total number of second-stage predicted speaker-changed tokens 302b in the second-stage sequence of predicted speaker-changed tokens 302b. In other words, the token-level loss function module 440 determines the precision metric 442 based on the ratio of the number of second-stage predicted speaker-changed tokens 302b labeled as correct to the total number of second-stage predicted speaker-changed tokens 302b in the second-stage sequence of predicted speaker-changed tokens 302b. Although not shown in FIG. 4, the training process 400 can train the sequence transformation model 300 by updating the parameters of the sequence transformation model 300 based directly on the accuracy metric 442 .

[0042] In some implementations, the labeler 430 labels each ground truth speaker change interval 414. That is, the labeler 430 generates interval tokens 434 indicating a correct match when any of the second-stage predicted speaker change tokens 302b overlap with the corresponding ground truth speaker change interval 434. Conversely, the labeler 430 generates interval tokens 434 indicating an incorrect match when none of the second-stage predicted speaker change tokens 302b overlap with the corresponding ground truth speaker change interval 434. A token-level loss function 440 receives the interval tokens 434 generated by the labeler 430 for each multi-utterance training sample 410. In some examples, the token-level loss function 440 determines a recall metric 444 for the sequence transfer model 300 based on the number of ground truth speaker change intervals 414 labeled as correct matches and the total number of all ground truth speaker change intervals 414. In another example, the token-level loss function 440 determines a recall metric 444 for the sequence transfer model 300 based on the duration of the ground truth speaker change intervals labeled as correctly matched and the total duration of all ground truth speaker change intervals. Determining the recall metric based on duration gives priority to multi-utterance training samples 410 with longer speaker change intervals. Although not shown in FIG. 4, the training process 400 can train the sequence transfer model 300 by updating its parameters directly based on the recall metric 444.

[0043]

number

[0044]

number

[0045] Advantageously, in some examples, the training process 400 determines the accuracy metric 442 of the sequence-transfer model 300 by determining an accuracy metric 442 that is not based on any word-level speech recognition results (e.g., transcriptions 342) output by the sequence-transfer model 300. In other examples, determining the accuracy metric 442 does not require the training process 400 to perform full speaker diarization on the audio data 412. Instead, the training process 400 can simply output the predicted speaker-change tokens 302 and transcriptions 342 without assigning a speaker label to each frame or token of the transcriptions 342.

[0046] 6 includes a flowchart of an exemplary arrangement of operations of a computer-implemented method for evaluating speaker change detection in a continuous multi-speaker conversational input audio stream. Method 600 can be executed on data processing hardware 710 (FIG. 7) using instructions stored in memory hardware 720 (FIG. 7), which may reside on user device 110 and / or cloud computing environment 140 of FIG. 1, corresponding to computing device 700 (FIG. 7).

[0047] At operation 602, the method 600 includes obtaining a multi-utterance training sample 410 including audio data 412 characterizing utterances spoken by two or more different speakers 10. At operation 604, the method 600 includes obtaining ground truth speaker change intervals 414 indicating time intervals in the audio data 412 during which speaker changes between the two or more different speakers occur. At operation 606, the method 600 includes processing the audio data 412 using the sequence transformation model 300 to generate a sequence of predicted speaker change tokens 302, each indicating the location of a speaker change within the audio data. At operation 608, for each corresponding predicted speaker change token 302, the method 600 includes labeling the corresponding predicted speaker change token 302 as correct when the predicted speaker change token 302 overlaps with one of the ground truth speaker change intervals 414. At operation 610, the method 600 includes determining an accuracy metric 442 for the sequence transfer model 300 based on the number of predicted speaker change tokens 302 labeled as correct and the total number of predicted speaker change tokens 302 in the sequence of predicted speaker change tokens 302. In some examples, the method 600 includes determining a recall metric 444 (additionally or alternatively, the accuracy metric 442) for the sequence transfer model 300 based on the durations of the ground truth speaker changes 414 labeled as correctly matched intervals and the total duration of all ground truth speaker change intervals 414.

[0048] 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. Computing device 700 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are intended to be exemplary only and are not intended to limit the scope of the embodiments of the invention described and / or claimed herein.

[0049] Computing device 700 includes a processor 710, memory 720, a storage device 730, a high-speed interface / controller 740 connecting to memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connecting to a low-speed bus 770 and storage device 730. Each of components 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as desired. Processor 710 is capable of processing instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730, for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to high-speed interface 740. In other embodiments, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as desired. Additionally, multiple computing devices 700 may be connected together (eg, as a bank of servers, a group of blade servers, or a multi-processor system) with each device providing a portion of the required operations.

[0050] Memory 720 stores information non-temporarily within computing device 700. Memory 720 may be a computer-readable medium, volatile memory unit(s), or non-volatile memory unit(s). Non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0051] Storage device 730 can provide mass storage for computing device 700. In some embodiments, storage device 730 is a computer-readable medium. In various different implementations, storage device 730 may be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In further embodiments, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 720, storage device 730, or memory on processor 710.

[0052] The high-speed controller 740 manages the bandwidth-intensive operations of the computing device 700, while the low-speed controller 760 manages the bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, the high-speed controller 740 is coupled to the memory 720, the display 780 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 750, which can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to the storage device 730 and the low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or networking device, such as a switch or router, via, for example, a network adapter.

[0053] Computing device 700 may be implemented in a number of different forms, as shown in the figure: for example, it may be implemented as a standard server 700a or as a repetition within a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0054] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general-purpose, and may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from, and transmit data and instructions to, the storage system.

[0055] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0056] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose processors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0057] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer that has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, speech input, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0058] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.< / st> < / st> < / st>

Claims

1. 1. A computer-implemented method (600) that, when executed on data processing hardware (710), causes the data processing hardware (710) to perform operations for evaluating detection of speaker changes in a continuous multi-speaker conversational input audio stream, the operations comprising: Obtaining a multi-utterance training sample (410) including audio data (412) characterizing utterances spoken by two or more different speakers (10); obtaining ground truth speaker change intervals (414) indicating time intervals in the audio data (412) during which speaker changes between the two or more different speakers (10) occur; processing the audio data (412) using a sequence transformation model (300) to generate a sequence of predicted speaker change tokens (302), each indicating the location of each speaker change within the audio data (412); for each corresponding predicted speaker-changed token (302), labeling the corresponding predicted speaker-changed token (302) as correct when the predicted speaker-changed token (302) overlaps with one of the ground truth speaker-change intervals (414); determining an accuracy metric (442) for the sequence transfer model (300) based on the number of the predicted speaker-change tokens (302) labeled as correct and the total number of the predicted speaker-change tokens (302) in the sequence of the predicted speaker-change tokens (302); A computer-implemented method (600) comprising:

2. 2. The computer-implemented method of claim 1, wherein determining the accuracy metric of the sequence transfer model is based on a ratio of the number of predicted speaker change tokens labeled as correct to the total number of predicted speaker change tokens in the sequence of predicted speaker change tokens.

3. 6. The computer-implemented method of claim 1, wherein the operations further comprise, for each corresponding predicted speaker-change token, labeling the corresponding predicted speaker-change token as a false-accept prediction when the corresponding predicted speaker-change token does not overlap with any of the ground truth speaker-change intervals.

4. The operation is for each ground truth speaker change interval (414), when any of the predicted speaker change tokens (302) overlaps with the corresponding ground truth speaker change interval (414), labeling the corresponding ground truth speaker change interval (414) as a correct match; determining a recall metric (444) for the sequence transfer model (300) based on the durations of the ground truth speaker change intervals (414) labeled as correctly matched and the total duration of all the ground truth speaker change intervals (414); 6. The computer-implemented method (600) of any preceding claim, further comprising:

5. 5. The computer-implemented method of claim 4, wherein the operations further comprise determining a performance score for the sequence transformation model based on the precision metric and the recall metric.

6. 6. The computer-implemented method of claim 5, wherein determining the performance score includes calculating the performance score based on the formula: 2×(precision metric (442)×recall metric (444)) / (precision metric (442)+recall metric (444)).

7. The operation is for each ground truth speaker change interval (414), when any of the predicted speaker change tokens (302) overlaps with the corresponding ground truth speaker change interval (414), labeling the corresponding ground truth speaker change interval (414) as a correct match; determining a recall metric (444) for the sequence transfer model (300) based on the number of the ground truth speaker change segments (414) labeled as correctly matched and the total number of all the ground truth speaker change segments (414); 6. The computer-implemented method (600) of any preceding claim, further comprising:

8. 8. The computer-implemented method of claim 7, wherein the operations further comprise determining a performance score for the sequence transformation model based on the precision metric and the recall metric.

9. 10. The computer-implemented method of claim 9, wherein determining the performance score comprises calculating the performance score based on the formula: 2×(Precision Metric (442)×Recall Metric (444)) / (Precision Metric (442)+Recall Metric (444)).

10. the multiple utterance training samples (410) further include ground truth speaker labels (418) paired with the audio data (412), each of the ground truth speaker labels (418) indicating a corresponding time-stamped segment in the audio data (412) associated with each of the utterances spoken by one of the two or more different speakers (10); Obtaining the ground truth speaker change interval (414) includes: identifying each time interval where two or more of the time-stamped segments overlap as a corresponding ground truth speaker change interval (414); identifying respective time gaps indicating pauses between two adjacent time-stamped segments in the audio data (412) associated with each of the utterances spoken by two different speakers (10); Including, A computer-implemented method (600) according to any of the preceding claims.

11. The operation is determining minimum and maximum start times for the audio data (412) based on the time-stamped segments indicated by the ground truth speaker labels (418); excluding predicted speaker change tokens (300) having a timestamp earlier than the minimum start time or later than the maximum start time from the determination of the accuracy metric (442) of the sequence transformation model (300); 11. The computer-implemented method (600) of claim 10, further comprising:

12. 10. The computer-implemented method of claim 9, wherein determining the accuracy metric of the sequence-to-sequence model is not based on any word-level speech recognition results output by the sequence-to-sequence model.

13. 10. The computer-implemented method (600) of any preceding claim, wherein said determining said accuracy metric (442) does not require performing a full speaker diarization on said audio data (412).

14. A system (100), comprising: Data processing hardware (710); and memory hardware (720) in communication with the data processing hardware (710), the memory hardware (720) storing instructions that, when executed by the data processing hardware (710), cause the data processing hardware (710) to perform operations, the operations including: Obtaining a multi-utterance training sample (410) including audio data (412) characterizing utterances spoken by two or more different speakers (10); obtaining ground truth speaker change intervals (414) indicating time intervals in the audio data (412) during which speaker changes between the two or more different speakers (10) occur; processing the audio data (412) using a sequence transformation model (300) to generate a sequence of predicted speaker change tokens (302), each indicating the location of each speaker change within the audio data (412); for each corresponding predicted speaker-changed token (302), labeling the corresponding predicted speaker-changed token (302) as correct when the predicted speaker-changed token (302) overlaps with one of the ground truth speaker-change intervals (414); determining an accuracy metric (442) for the sequence transfer model (300) based on the number of the predicted speaker-change tokens (302) labeled as correct and the total number of the predicted speaker-change tokens (302) in the sequence of the predicted speaker-change tokens (302); A system (100) comprising:

15. 15. The system of claim 14, wherein determining the accuracy metric of the sequence transformation model is based on a ratio of the number of the predicted speaker change tokens labeled as correct to the total number of the predicted speaker change tokens in the sequence of predicted speaker change tokens.

16. 16. The system (100) of claim 14, wherein the operations further include, for each corresponding predicted speaker-change token (302), labeling the corresponding predicted speaker-change token (302) as a false-accept prediction when the corresponding predicted speaker-change token (302) does not overlap with any of the ground truth speaker-change intervals (414).

17. The operation is for each ground truth speaker change interval (414), when any of the predicted speaker change tokens (302) overlaps with the corresponding ground truth speaker change interval (414), labeling the corresponding ground truth speaker change interval (414) as a correct match; determining a recall metric (444) for the sequence transfer model (300) based on the durations of the ground truth speaker change intervals (414) labeled as correctly matched and the total duration of all the ground truth speaker change intervals (414); The system (100) of any of claims 14 to 16, further comprising:

18. 20. The system of claim 17, wherein the operations further comprise determining a performance score for the sequence transformation model based on the precision metric and the recall metric.

19. 20. The system of claim 18, wherein determining the performance score includes calculating the performance score based on the formula: 2 x (precision metric (442) x recall metric (444)) / (precision metric (442) + recall metric (444)).

20. The operation is for each ground truth speaker change interval (414), when any of the predicted speaker change tokens (302) overlaps with the corresponding ground truth speaker change interval (414), labeling the corresponding ground truth speaker change interval (414) as a correct match; determining a recall metric (444) for the sequence transfer model (300) based on the number of the ground truth speaker change segments (414) labeled as correctly matched and the total number of all the ground truth speaker change segments (414); The system (100) of any of claims 14 to 19, further comprising:

21. 21. The system of claim 20, wherein the operations further comprise determining a performance score for the sequence transformation model based on the precision metric and the recall metric.

22. 22. The system (100) of claim 14, wherein determining the performance score (452) comprises calculating the performance score (452) based on the formula: 2×(precision metric (442)×recall metric (444)) / (precision metric (442)+recall metric (444)).

23. the multiple utterance training samples (410) further include ground truth speaker labels (418) paired with the audio data (412), each of the ground truth speaker labels (418) indicating a corresponding time-stamped segment in the audio data (412) associated with each of the utterances spoken by one of the two or more different speakers (10); Obtaining the ground truth speaker change interval (414) includes: identifying each time interval where two or more of the time-stamped segments overlap as a corresponding ground truth speaker change interval (414); identifying respective time gaps indicating pauses between two adjacent time-stamped segments in the audio data (412) associated with each of the utterances spoken by two different speakers (10); Including, A system (100) according to any one of claims 14 to 22.

24. The operation is determining minimum and maximum start times for the audio data (412) based on the time-stamped segments indicated by the ground truth speaker labels (418); excluding predicted speaker change tokens (300) having a timestamp earlier than the minimum start time or later than the maximum start time from the determination of the accuracy metric (442) of the sequence transformation model (300); 24. The system (100) of claim 23, further comprising:

25. 25. The system of claim 14, wherein determining the accuracy metric of the sequence-to-sequence model is not based on any word-level speech recognition results output by the sequence-to-sequence model.

26. The system (100) of any of claims 14 to 25, wherein said determining said accuracy metric (442) does not require performing a full speaker diarization on said audio data (412).

Images