Multi-speaker audio stream separation, transcription and digitization

Abstract

A plurality of speaker embedding vectors corresponding to a plurality of speakers in an input audio stream may be derived. Each speaker embedded vector may represent a respective speech feature of a respective speaker. The speaker embedding vector may be generated based on, for example, a pre-registration process or a cluster-based embedding vector derivation process. Multiple instances of a personalized noise suppression model may be performed on an input audio stream. Each instance of the personalized noise suppression model may employ a respective speaker embedding vector. Multiple single speaker audio streams may be generated from multiple instances of a personalized noise suppression model. A plurality of single speaker transcriptions may be generated based on the plurality of single speaker audio streams. The plurality of single speaker transcriptions may be merged into a multi-speaker output transcription.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 850,617, filed on June 27, 2022, the entire contents of which are incorporated herein by reference. Background Art

[0003] Automatic speech transcription is a valuable feature for archiving and accessibility. For example, using speech transcription can allow recordings of meetings, performances, or other events to be generated and saved. Additionally, in some examples, speech transcription can allow textual indications of speech to be provided to audiences who may be hard of hearing or in situations where the listener cannot clearly hear the speech. In some cases, such as in certain noisy conditions or multi-speaker scenarios, the performance of speech transcription systems may degrade. Some speech transcription systems may be trained for situations where a single speaker is active or where there is clear turn-taking between multiple speakers. Therefore, the performance of some speech transcription systems may degrade in some situations where multiple speakers are active at the same time. Additionally, if the transcription system is also configured for journaling, speaker labeling may be affected when multiple speakers are active at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The following detailed description may be better understood when read in conjunction with the accompanying drawings.For purposes of illustration, example embodiments of various aspects of the disclosure are shown in the drawings; however, the invention is not limited to the specific methods and instrumentalities disclosed.

[0005] Figure 1 is a diagram illustrating an example multi-speaker audio stream separation, journaling, and transcription system that may be used in accordance with the present disclosure.

[0006] Figure 2 is a diagram illustrating an exemplary pre-enrollment embedding vector derivation that may be used in accordance with the present disclosure.

[0007] Figure 3 is a diagram illustrating an exemplary post-completion cluster-based embedding vector derivation that may be used in accordance with the present disclosure.

[0008] Figure 4 is a diagram illustrating an exemplary in-progress cluster-based embedding vector derivation that may be used in accordance with the present disclosure.

[0009] Figure 5 is a diagram illustrating exemplary clustering of multi-speaker embedding vectors with discard that may be used in accordance with the present disclosure.

[0010] Figure 6 is a flow chart illustrating an exemplary multi-speaker audio stream separation, journaling, and transcription process that may be used in accordance with the present disclosure.

[0011] Figure 7 is a flow chart illustrating an exemplary cluster-based embedding vector derivation process that may be used in accordance with the present disclosure.

[0012] Figure 8 is a diagram illustrating an example system for transmitting and providing data that may be used in accordance with the present disclosure.

[0013] Fig. 9 is a diagram illustrating an example computing system that can be used in accordance with the present disclosure. DETAILED DESCRIPTION

[0014] This article describes techniques for multi-speaker audio stream separation, transcription, and diary. An input audio stream may include the speech of multiple speakers. According to the techniques described herein, multiple speaker embedding vectors may be derived, each speaker embedding vector corresponding to a corresponding speaker among the multiple speakers in the input audio stream. Each speaker embedding vector may represent a corresponding voice feature of the corresponding speaker. In addition, multiple instances of a personalized noise suppression model may be executed on the input audio stream. Specifically, each instance of the personalized noise suppression model may correspond to a corresponding speaker among the multiple speakers, and a corresponding speaker embedding vector may be adopted for its corresponding speaker. The personalized noise suppression model may be a machine learning model that is trained to retain a specific voice represented by an embedding vector. Therefore, each instance of the personalized noise suppression model may use its corresponding embedding vector to retain the speech corresponding to its corresponding speaker and suppress any signal other than the speech of the corresponding speaker.

[0015] Multiple instances of the personalized noise suppression model may be executed on an input audio stream to generate multiple single-talker audio streams. Each of the multiple single-talker audio streams may be generated by a corresponding instance of the multiple instances of the personalized noise suppression model by outputting only sounds from the input audio stream corresponding to the corresponding speaker embedding vector. Thus, each single-talker audio stream may include speech only from the corresponding speaker, without speech from other speakers. The multiple single-talker audio streams may be generated using a consistent time representation scheme (e.g., consistent timestamps) that is consistent across the multiple single-talker audio streams.

[0016] Multiple single-speaker audio streams may be provided to a transcription system, which may generate multiple single-speaker transcriptions from the multiple single-speaker audio streams. Each single-speaker transcription in the multiple single-speaker transcriptions may be generated from a corresponding single-speaker audio stream in the multiple single-speaker audio streams. Thus, each single-speaker transcription may be a transcription of speech only from the corresponding speaker, and not from speech from other speakers. Each single-speaker transcription may be generated using a consistent time representation scheme of the underlying single-speaker audio stream. The multiple single-speaker transcriptions may then be merged into a multi-talker output transcription corresponding to the input audio stream based on the consistent time representation scheme.

[0017] In some examples, multiple speaker embedding vectors may be derived based on a pre-registration process for multiple speakers. The term pre-registration as used herein refers to a registration process for multiple speakers, wherein multiple speaker embedding vectors are derived before initiating the generation of an input audio stream and / or before processing the input audio stream for transcription. In the pre-registration process, speech audio samples of each of the multiple speakers may be analyzed to derive multiple speaker embedding vectors. Specifically, a set of one or more audio samples may be provided for each speaker, and a set of one or more audio samples may include only the speech of the speaker, without including the speech of other speakers. Speaker embedding vectors for the corresponding speakers may then be derived based on the set of one or more audio samples. In some examples, identity data indicating the identity of the speaker may be associated with a corresponding set of one or more audio samples and a corresponding speaker embedding vector derived from the corresponding set of one or more audio samples. The identity data may allow the identity of the speaker to be indicated in association with the transcribed speech from the speaker in the speaker's respective single-speaker audio stream and multi-speaker output transcription. In some cases, the pre-registration process may be particularly suitable for online applications that employ live transcription, as it allows multiple speaker embedding vectors to be derived before generating the input audio stream.

[0018] In some other examples, multiple speaker embedding vectors can be derived based on cluster-based embedding vector derivation. In cluster-based embedding vector derivation, speaker embedding vectors are derived based on the input audio stream itself. Therefore, unlike the pre-registration process, cluster-based embedding vector derivation does not allow multiple speaker embedding vectors to be derived before starting the generation of the input audio stream. In contrast, in cluster-based embedding vector derivation, multiple speaker embedding vectors are derived after the input audio stream is completely generated (hereinafter referred to as completed cluster-based embedding vector derivation) or while the input audio stream generation is in progress (hereinafter referred to as ongoing cluster-based embedding vector derivation).

[0019] During the post-cluster-based embedding vector derivation, the speaker enumeration machine learning model can analyze the entire input audio stream to estimate the time-varying number of concurrent speakers within the input audio stream at multiple time points. The time-varying number of concurrent speakers can then be used to determine multiple single-speaker segments in the input audio stream with only one speaker. Multiple segment embedding vectors can then be derived from the multiple single-speaker segments, wherein one or more segment embedding vectors in the multiple segment embedding vectors are derived for each single-speaker segment in the multiple single-speaker segments. The multiple segment embedding vectors can then be clustered into multiple embedding vector clusters. Each embedding vector cluster can correspond to a corresponding speaker in the multiple speakers. Multiple representative embedding vectors can then be derived from the multiple embedding vector clusters. Each representative embedding vector in the multiple representative embedding vectors can be derived for a corresponding embedding vector cluster in the multiple embedding vector clusters. The multiple representative embedding vectors can then be used as multiple speaker embedding vectors. Since the post-cluster-based embedding vector derivation is performed after the input audio stream is fully generated, the post-cluster-based embedding vector derivation may be particularly suitable for offline non-live applications.

[0020] The ongoing cluster-based embedding vector derivation may be similar to the post-completion cluster-based embedding vector derivation, but with some exceptions. Specifically, although the post-completion cluster-based embedding vector derivation can be performed on the entire input audio stream after the input audio stream is fully generated, the ongoing cluster-based embedding vector derivation can be repeatedly performed in a running fashion during the generation of the input audio stream. Specifically, the ongoing cluster-based embedding vector derivation can be repeated on different parts of the input audio stream in multiple iterations when different parts of the input audio stream are generated. Therefore, for the ongoing cluster-based embedding vector derivation, the number of speakers and speaker clustering can be estimated in a running fashion. In addition, for the ongoing cluster-based embedding vector derivation, speaker enumeration, speaker clustering, and representative speaker embedding can be continuously improved (e.g., in each iteration of the derivation) so that speech extraction performance improves as the stream progresses.

[0021] Thus, by separating an input audio stream into multiple single-speaker audio streams, generating single-speaker transcripts, and merging the single-speaker audio transcripts into a multi-speaker output transcript, the techniques described herein allow for multi-speaker speech separation, transcription, and diarying. Notably, the techniques described herein are particularly advantageous for speech transcription and diarying in the presence of background speech (e.g., call centers, events with audiences, etc.). The techniques described herein are also particularly advantageous for speech transcription and diarying in situations where multiple speakers are active simultaneously.

[0022] Figure 1is a diagram illustrating an example multi-speaker audio stream separation, diary, and transcription system that may be used in accordance with the present disclosure. Figure 1 In the example of , the input audio stream 100 includes speech from three speakers, including Ann, Bob, and Carol. The input audio stream 100 may include audio from a variety of sources, including, for example, online conferences, telephone calls, call centers, movies, television programs, events (e.g., sporting events, news events, entertainment events, performances, etc.), and / or other sources. In some examples, the input audio stream 100 may be a live stream that is captured and played to the audience without a noticeable delay between the time the stream is captured and the time the stream is played to the audience. In some other examples, the input audio stream 100 may be a pre-recorded stream that is recorded in advance, saved, and then played to the audience later after the input audio stream has been fully generated.

[0023] As will be described in detail below, speaker embedding vectors (TEVs) 105A-C may be derived for speakers in the input audio stream 100. In this example, speaker embedding vector (TEV) 105A corresponds to Ann, TEV 105B corresponds to Bob, and TEV 105C corresponds to Carol. Each TEV 105A-C may represent a corresponding voice feature of a corresponding speaker. Specifically, TEV 105A represents the voice features of Ann, TEV 105B represents the voice features of Bob, and TEV 105C represents the voice features of Carol. In addition, in this example, the stream separation component 150 performs personalized noise suppression (PNS) instances 101A-C on the input audio stream 100. PNS instances 101A-C are instances of personalized noise suppression models. Each PNS instance 101A-C corresponds to a corresponding speaker among a plurality of speakers in the input audio stream 100. Specifically, PNS instance 101A corresponds to Ann, PNS instance 101B corresponds to Bob, and PNS instance 101C corresponds to Carol. In addition, each PNS instance 101A-C adopts one of the TEVs 105A-C for its corresponding speaker. Specifically, PNS instance 101A adopts TEV 105A, PNS instance 101B adopts TEV 105B, and PNS instance 101C adopts TEV 105C. The personalized noise suppression model can be a machine learning model that is trained to preserve the specific voice represented by the embedding vector. Therefore, each PNS instance 101A-C can respectively adopt TEV 105A-C to preserve the speech corresponding to its respective speaker and suppress any signal other than the speech of the corresponding speaker.

[0024] like Figure 1As shown, PNS instances 101A-C may be executed on input audio stream 100 to generate single-speaker audio streams 102A-C. Specifically, single-speaker audio stream 102A corresponds to Ann, single-speaker audio stream 102B corresponds to Bob, and single-speaker audio stream 102C corresponds to Carol. Each single-speaker audio stream 102A-C is generated by a corresponding PNS instance in PNS instances 101A-C by outputting only sounds corresponding to corresponding TEVs in TEVs 105A-C from the input audio stream 100. Therefore, each single-speaker audio stream 102A-C may include only speech from the corresponding speaker, and not speech from other speakers. Specifically, single-speaker audio stream 102A may include only speech from Ann, single-speaker audio stream 102B may include only speech from Bob, and single-speaker audio stream 102C may include only speech from Carol. The single-speaker audio streams 102A-C may be generated using a consistent time representation scheme (eg, consistent timestamps) that is consistent across the single-speaker audio streams 102A-C.

[0025] Likewise Figure 1 As shown, the single-speaker audio streams 102A-C may be provided to the transcription component 160, and the transcription component 160 may generate single-speaker transcriptions 103A-C from the single-speaker audio streams 102A-C. Each single-speaker transcription 103A-C is generated from a corresponding single-speaker audio stream in the single-speaker audio streams 102A-C. Therefore, each single-speaker transcription 103A-C may be a transcription of speech only from the corresponding speaker and not other speakers. Specifically, the single-speaker transcription 103A may be a transcription of speech only from Alice, the single-speaker transcription 103B may be a transcription of speech only from Bob, and the single-speaker transcription 103C may be a transcription of speech only from Carol. Each of the single-speaker transcriptions 103A-C may be generated using a consistent time representation scheme of the underlying single-speaker audio streams. For example, each of the single-speaker transcriptions 103A-C may include an indication of the words spoken by the respective speakers and an indication of the time at which the words were spoken.

[0026] The transcription component 160 may then merge the single-speaker transcriptions 103A-C into a merged transcription 104. The merged transcription 104 is a multi-speaker output transcription corresponding to the input audio stream 100. The single-speaker transcriptions 103A-C may be merged into the merged transcription 104 based on a consistent time representation scheme. For example, the merged transcription 104 may include an indication of the words spoken by each speaker (e.g., Alice, Bob, and Carol) in the input audio stream 100, an indication of the time at which the words were spoken, and an indication of the identity of the speaker who spoke the words. In some examples, the identity of the speaker may be determined based on the single-speaker transcriptions 103A-C from which the words were obtained. For example, it may be determined that the words obtained from the single-speaker transcription 103A were spoken by Ann, the words obtained from the single-speaker transcription 103B were spoken by Bob, and the words obtained from the single-speaker transcription 103C were spoken by Carol.

[0027] Reference now Figures 2 to 4 , shows some examples of techniques that can be used to generate TEV 105A-C. These example techniques can include pre-registered embedding vector derivation (see, e.g. Figure 2 ), after completion, cluster-based embedding vectors are derived (see e.g. Figure 3 ) and ongoing cluster-based embedding vector derivation (see e.g. Figure 4 ). Now refer to Figure 2 , an example of pre-registration embedding vector (EV) derivation 200 will now be described. The term "pre-registration" as used herein refers to a registration process for multiple speakers, wherein multiple speaker embedding vectors (e.g., TEVs 105A-C) are derived before starting to generate the input audio stream 100 and / or before processing the input audio stream 100 for transcription. During the pre-registration embedding vector (EV) derivation 200, audio samples 201A-C of each speaker's speech may be analyzed to derive multiple TEVs 105A-C. In this example, audio sample 201A includes only Ann's speech samples (no other speakers), audio sample 201B includes only Bob's speech samples (no other speakers), and audio sample 201C includes only Carol's speech samples (no other speakers). TEV 105A may be derived based on audio sample 201A, TEV 105B may be derived based on audio sample 201B, and TEV 105C may be derived based on audio sample 201C. After the pre-registered embedding vector (EV) is derived 200, it can then be Figure 1 The input audio stream is shown being processed by having PNS instances 101A-C analyze the input audio stream 100 using TEVs 105A-C, respectively.

[0028] In some examples, as part of the preregistered embedding vector (EV) derivation 200, identity data indicating the identities of the respective speakers may be associated with the audio samples 201A-C and the TEVs 105A-C derived from the audio samples 201A-C. For example, identity data indicating the identity of Ann may be associated with the audio sample 201A and the TEV 105A, identity data indicating the identity of Bob may be associated with the audio sample 201B and the TEV 105B, and identity data indicating the identity of Carol may be associated with the audio sample 201C and the TEV 105C. The identity data may allow the identities of the speakers to be indicated in association with the transcribed speech from the speakers in the single speaker transcriptions 103A-C and the merged transcription 104. In some cases, the preregistered embedding vector (EV) derivation 200 may be particularly suitable for online applications that employ live transcription because it allows the TEVs 105A-C to be derived before the input audio stream 100 is generated.

[0029] In some other examples, the TEVs 105A-C may be derived based on cluster-based embedding vector derivation. In cluster-based embedding vector derivation, the TEVs 105A-C are derived based on the input audio stream 100 itself. Figure 2 Unlike the pre-registered embedding vector (EV) derivation 200 of the present invention, the cluster-based embedding vector derivation does not allow the TEV 105A-C to be derived before starting the generation of the input audio stream 100. Instead, in the cluster-based embedding vector derivation, the TEV 105A-C is derived after the input audio stream is completely generated (hereinafter referred to as post-completed cluster-based embedding vector derivation) or while the generation of the input audio stream is ongoing (hereinafter referred to as ongoing cluster-based embedding vector derivation).

[0030] Reference now Figure 3 , we will now describe an example of cluster-based embedding vector derivation after completion. Figure 3 As shown, after completing the cluster-based embedding vector derivation, the speaker enumeration machine learning (ML) model 300 can analyze the entire input audio stream 100 to estimate the time-varying number of co-existing speakers within the input audio stream 100 at multiple time points. Specifically, the speaker enumeration ML model 300 can estimate the number of speakers who are speaking within the input audio stream 100 at any given time. The time-varying number of co-existing speakers can then be used to determine single speaker segments (STS) 311-316 of the input audio stream 100. Each single speaker segment (STS) 311-316 is a segment in the input audio stream 100 with only one speaker. Embedding vectorization 310 can then be performed on each STS311-316 to derive a segment embedding vector (SEV) 301-308. Each STS 311-316 includes at least one of the SEVs301-308. In Figure 3 In the example of , SEV 301 and 302 are derived for STS 311, segment embedding vector (SEV) 303 is derived for STS 312, SEV 304 and 305 are derived for STS 313, SEV 306 is derived for STS 314, SEV 307 is derived for STS 315, and SEV 308 is derived for STS 316. In this example, each SEV 301-308 is an embedding vector that represents the audio characteristics of a person speaking in a corresponding STS in STS 311-316. For example, SEV 301 and 302 may represent the audio characteristics of a person speaking in STS 311. SEV 301 may be generated based on speech in the first half of STS 311, and SEV 302 may be generated based on speech in the second half of STS 312. Thus, even though they may represent the voice of the same person, SEV 301 and SEV 302 may differ slightly from one another because the characteristics of a person's voice (e.g., pitch, volume / amplitude, speaking speed, etc.) may vary slightly over time.

[0031] Embedding vector clustering algorithm 320 may then be performed on SEVs 301-308 to cluster SEVs 301-308 into embedding vector clusters 330A, 330B, and 330C. In this example, embedding vector cluster 330A includes SEVs 301, 302, and 306, embedding vector cluster 330B includes SEVs 304, 305, and 307, and embedding vector cluster 330C includes SEVs 303 and 308. Embedding vector clustering algorithm 320 may cluster SEVs 301-308 into embedding vector clusters 330A-C based on relative similarities between SEVs 301-308. For example, SEVs 301, 302, and 306 clustered into embedding vector cluster 330A may have greater similarity to each other than to any of SEVs 303-305, 307, and 308 that are not included in embedding vector cluster 330A. Similarly, SEVs 304, 305, and 307 clustered into embedding vector cluster 330B may have greater similarity to each other than to any of SEVs 301-303, 306, and 308 not included in embedding vector cluster 330B. Furthermore, SEVs 303 and 308 clustered into embedding vector cluster 330C may have greater similarity to each other than to any of SEVs 301-302 and 304-307 not included in embedding vector cluster 330C.

[0032] Each embedding vector cluster 330A-C can correspond to a corresponding speaker of the plurality of speakers. Specifically, embedding vector cluster 330A corresponds to Ann, embedding vector cluster 330B corresponds to Bob, and embedding vector cluster 330C corresponds to Carol. In this example, since there are three speakers (Ann, Bob, and Carol) in the input audio stream 100, three embedding vector clusters 330A-C are formed. It should be noted that for cluster-based embedding vector derivation, the number of speakers in the input audio stream 100 (and the number of embedding vector clusters 330A-C formed by clustering) is not known in advance. Instead, the number of speakers and embedding vector clusters 330A-C are determined based on the results of clustering, such as based on the relative similarities between SEVs 301-308.

[0033] During representative embedding vector (REV) derivation 340, representative embedding vectors (REV) 331-333 may be derived from embedding vector clusters 330A-C. Specifically, REV 331A is derived from embedding vector cluster 330A, REV 331B is derived from embedding vector cluster 330B, and REV 331C is derived from embedding vector cluster 330C. REV 331A is an embedding vector representing SEVs 301, 302, and 306 within embedding vector cluster 330A. For example, REV 331A may be the centroid of SEVs 301, 302, and 306, or may be generated based on the centroid of SEVs 301, 302, and 306. REV 331B is an embedding vector representing SEVs 304, 305, and 307 in embedding vector cluster 330B. For example, REV 331B may be the centroid of SEVs 304, 305, and 307, or may be generated based on the centroids of SEVs 304, 305, and 307. REV 331C is an embedding vector representing SEVs 303 and 308 in embedding vector cluster 330C. For example, REV 331C may be the centroid of SEVs 303 and 308, or may be generated based on the centroids of SEVs 303 and 308. Then, REVs 331A-C may be used as TEVs 105A-C for respective speakers. Specifically, REV 331A may be used as TEV 105A for Ann, REV 331B may be used as TEV 105B for Bob, and REV 331C may be used as TEV 105C for Carol. Since the post-completion cluster-based embedding vector derivation is performed after the input audio stream 100 is completely generated, the post-completion cluster-based embedding vector derivation may be particularly suitable for offline non-live applications.

[0034] Reference now Figure 4 , an example of ongoing cluster-based embedding vector derivation will now be described in detail. Figure 4In the example of FIG. 4 , ongoing cluster-based embedding vector derivation is performed on an input audio stream 400, which also includes audio of speakers Ann, Bob, and Carol. The input audio stream 400 includes stream portions 400A-B and optionally any number of additional stream portions ( Figure 4 ). The ongoing cluster-based embedding vector derivation may be similar to the post-completion cluster-based embedding vector derivation, but with some exceptions. Specifically, although the post-completion cluster-based embedding vector derivation can be performed on the entire input audio stream 100 after the input audio stream 100 is fully generated, the ongoing cluster-based embedding vector derivation can be repeatedly performed in a flowing manner when the input audio stream 400 is generated. Specifically, when different parts of the input audio stream 400 (e.g., stream parts 400A-B) are generated, the ongoing cluster-based embedding vector derivation can be repeatedly performed on different parts of the input audio stream 400 (e.g., stream parts 400A-B) in multiple iterations (e.g., cluster-based EV derivation iterations 401-402). Therefore, for the ongoing cluster-based embedding vector derivation, the number of speakers and speaker clustering can be estimated in a flowing manner. In addition, for the ongoing cluster-based embedding vector derivation, speaker enumeration, speaker clustering, and representative speaker embedding can be continuously improved (e.g., in each iteration of the derivation) so that the speech extraction performance improves as the stream progresses.

[0035] You can execute the above reference Figure 3 The actions described are performed Figure 4 4. In particular, each of the cluster-based EV derivation iterations 401-402 may include analyzing a corresponding stream portion in the stream portions 400A-B using the speaker enumeration ML model 300 to estimate a time-varying number of concurrent speakers within the corresponding stream portion in the stream portions 400A-B at multiple time points. The time-varying number of concurrent speakers may then be used to determine a plurality of single-speaker segments of the corresponding stream portion in the stream portions 400A-B in which there is only one speaker. During embedding vectorization 310, a plurality of segment embedding vectors may then be derived from the plurality of single-speaker segments. An embedding vector clustering algorithm 320 may then be performed to cluster the plurality of segment embedding vectors into one or more embedding vector clusters. Each embedding vector cluster may correspond to a corresponding speaker in the plurality of speakers. A representative embedding vector derivation 340 may then be performed to derive one or more representative embedding vectors from one or more embedding vector clusters. The one or more representative embedding vectors may then be used as one or more of the TEVs 105A-C.

[0036] For example, Figure 4As shown, cluster-based EV derivation iteration 401 is performed on stream portion 400A. In this example, stream portion 400A includes only one single speaker segment (STS), namely STS 411. As shown, STS 411 includes Ann's audio. Therefore, cluster-based EV derivation iteration 401 only provides representative embedding vector (REV) 431A-1, which is the initial REV corresponding to Ann. In contrast, stream portion 400B includes four STSs, namely STS 411-414. As shown, STS 411 includes Ann's audio, STS 412 includes Bob's audio, STS 413 includes Ann's additional audio, and STS 414 includes Carol's audio. Therefore, cluster-based EV derivation 402 provides REV 431A-2, REV 431B-1, and REV 431C-1. Specifically, REV431B-1 is the initial REV corresponding to Bob, REV 431C-1 is the initial REV corresponding to Carol, and REV 431A-2 is the improved REV corresponding to Ann. Therefore, although the cluster-based EV derivation iteration 401 only reflects a single speaker (Ann), the cluster-based EV derivation 402 reflects three speakers (Ann, Bob and Carol). In addition, the cluster-based EV derivation 402 also generates a more accurate representation of Ann's voice. Specifically, although REV 431A-1 is generated based only on STS 411 (corresponding to Ann), REV431A-2 is generated based on STS 411 and STS 413 (both corresponding to Ann). Therefore, since REV 431A-2 is generated based on a larger number of samples of Ann's voice, REV 431A-2 is an improved REV, which more accurately represents Ann's voice compared to REV 431A-1.

[0037] In the above Figure 3 In the example of , speaker enumeration ML model 300 is performed on the input audio stream to determine STS 311-316. Then, embedding vectorization 310 is performed only on STS 311-316 to determine SEV 301-308. However, in some other examples, speaker enumeration ML model 300 may not need to be performed before embedding vectorization 310. Now refer to Figure 5 , shows an example of an alternative technique where the speaker enumeration ML model 300 is not performed prior to embedding vectorization 310. Instead, Figure 5 In an alternative technique, embedding vectorization 310 is performed on the entire input audio stream 500 (or a portion thereof), rather than just on the single-speaker segments. In this alternative technique, embedding vectors corresponding to single-speaker segments are detected during the clustering process and then discarded, meaning that they are not used to determine the representative embedding vector.

[0038] like Figure 5 As shown, embedding vectorization 310 is performed on the input audio stream 500 to generate nine segment embedding vectors (SEVs) 501-509. Specifically, SEVs 501-503 correspond to the first portion of the input audio stream 500 in which only Ann is speaking. In addition, SEVs 504-506 correspond to the second portion of the input audio stream 500 in which both Ann and Bob are speaking. In addition, SEVs 507-509 correspond to the third portion of the input audio stream 500 in which only Bob is speaking.

[0039] In this example, embedding vector clustering algorithm 520 is performed on SEVs 501-509. Embedding vector clustering algorithm 520 clusters SEVs 501-503 into embedding vector cluster 530A corresponding to Ann. In addition, embedding vector clustering algorithm 520 clusters SEVs 507-509 into embedding vector cluster 530B corresponding to Bob. In addition, embedding vector clustering algorithm 520 detects that SEVs 504-506 correspond to portions of the input audio stream 500 where multiple speakers (e.g., Ann and Bob) speak simultaneously. Embedding vector clustering algorithm 520 groups SEVs 504-506 into discarded multi-speaker groups 535. Various techniques may be employed to detect that SEVs 504-506 correspond to portions of the input audio stream 500 where multiple speakers (e.g., Ann and Bob) speak simultaneously. In some examples, embedding vector clustering algorithm 520 may examine temporal (time-based) relationships between SEVs 501-509. Specifically, the embedding vector clustering algorithm 520 can examine a neighborhood of SEVs that includes two or more consecutive adjacent SEVs (or that are close in time to each other) to determine the amount of similarity or correlation between the adjacent SEVs. For example, for a scenario where only one person is speaking in adjacent SEVs, it can be expected that these adjacent SEVs have a high degree of similarity to each other because they correspond to the speech of the same person. In contrast, for a scenario where multiple speakers are speaking simultaneously in adjacent SEVs, it can be expected that these SEVs will not have a high degree of similarity because they correspond to the speech of more than one person. Figure 5 In the example of , it can be determined that adjacent SEVs 501-503 have a high degree of similarity to each other and therefore correspond to the same person (Ann). In addition, it can be determined that adjacent SEVs 507-509 also have a high degree of similarity to each other and therefore correspond to the same person (Bob). Conversely, it can be determined that adjacent SEVs 504-506 are not highly similar to each other and therefore correspond to a portion of the input audio stream 500 where multiple speakers (Ann and Bob) are speaking simultaneously.

[0040] like Figure 5As shown, a representative embedding vector (REV) 531A is determined for embedding vector cluster 530A and used as TEV 105A for Ann. In addition, REV 531B is determined for embedding vector cluster 530B and used as TEV 105B for Bob. However, it should be noted that REV is not determined for the discarded multi-speaker group 535. It should be noted that Figure 5 The alternative technique can be used both for post-completion cluster-based embedding vector derivation and for ongoing cluster-based embedding vector derivation.

[0041] Figure 6 6 is a flow chart illustrating an example multi-speaker audio stream separation, diaryization, and transcription process that may be used according to the present disclosure. At operation 610, a plurality of speaker embedding vectors are derived. The plurality of speaker embedding vectors may correspond to a plurality of speakers in an input audio stream. Each speaker embedding vector in the plurality of speaker embedding vectors may correspond to a respective speaker in the plurality of speakers and may represent a respective voice feature of the respective speaker. For example, Figure 1 As shown, the input audio stream 100 includes speech from three speakers, including Ann, Bob, and Carol. Speaker embedding vectors (TEVs) 105A-C can be derived for the speakers in the input audio stream 100. In this example, TEV 105A corresponds to Ann, TEV 105B corresponds to Bob, and TEV 105C corresponds to Carol. Each TEV 105A-C can represent the respective speech characteristics of each speaker. Specifically, TEV 105A represents the speech characteristics of Ann, TEV 105B represents the speech characteristics of Bob, and TEV 105C represents the speech characteristics of Carol.

[0042] Some example techniques for deriving TEV 105A-C are described in detail above and will not be repeated here. Specifically, deriving multiple speaker embedding vectors may include performing a registration process in which multiple speakers are registered, such as Figure 2 derive the pre-registered EV 200. In addition, deriving the plurality of speaker embedding vectors may include performing a cluster-based embedding vector derivation. In some examples, the cluster-based embedding vector derivation may be performed after the input audio stream is fully generated and may be performed on all input audio streams (e.g., Figure 3 In addition, in some examples, the cluster-based embedding vector derivation can be performed in a streaming manner during the generation of the input audio stream, and can be repeatedly performed on different parts of the input audio stream as different parts of the input audio stream are generated (e.g., Figure 4 In-progress cluster-based embedding vector derivation). In addition, Figure 5The alternative technique shown in can optionally be used for both post-completion cluster-based embedding vector derivation and ongoing cluster-based embedding vector derivation.

[0043] At operation 612, multiple instances of the personalized noise suppression model are executed on the input audio stream. Each instance of the multiple instances of the personalized noise suppression model uses a corresponding speaker embedding vector from the multiple speaker embedding vectors. For example, Figure 1 As shown, the stream separation component 150 performs personalized noise suppression (PNS) instances 101A-C on the input audio stream 100. PNS instances 101A-C are instances of personalized noise suppression models. Each PNS instance 101A-C corresponds to a corresponding speaker among multiple speakers in the input audio stream 100. Specifically, PNS instance 101A corresponds to Ann, PNS instance 101B corresponds to Bob, and PNS instance 101C corresponds to Carol. In addition, each PNS instance 101A-C adopts one of TEVs 105A-C for its corresponding speaker. Specifically, PNS instance 101A adopts TEV 105A, PNS instance 101B adopts TEV 105B, and PNS instance 101C adopts TEV 105C. The personalized noise suppression model can be a machine learning model that is trained to retain a specific voice represented by an embedding vector. Thus, each PNS instance 101A-C may respectively employ a TEV 105A-C to preserve the speech corresponding to its respective speaker and suppress any signals other than the speech of the respective speaker.

[0044] At operation 614, multiple single-speaker audio streams are generated by multiple instances of the personalized noise suppression model. The multiple single-speaker audio streams may correspond to the input audio stream. Each single-speaker audio stream in the multiple single-speaker audio streams may be generated by a respective instance in the multiple instances of the personalized noise suppression model by outputting only sounds from the input audio stream that correspond to the corresponding speaker embedding vector. The multiple single-speaker audio streams may be generated using a consistent time representation scheme that is consistent across the multiple single-speaker audio streams. For example, Figure 1As shown, PNS instances 101A-C may be executed on input audio stream 100 to generate single-speaker audio streams 102A-C. Specifically, single-speaker audio stream 102A corresponds to Ann, single-speaker audio stream 102B corresponds to Bob, and single-speaker audio stream 102C corresponds to Carol. Each single-speaker audio stream 102A-C is generated by a corresponding PNS instance in PNS instances 101A-C by outputting only the sound corresponding to the corresponding TEV in TEV 105A-C from the input audio stream 100. Therefore, each single-speaker audio stream 102A-C may only include speech from the corresponding speaker, and not speech from other speakers. Specifically, single-speaker audio stream 102A may only include speech from Ann, single-speaker audio stream 102B may only include speech from Bob, and single-speaker audio stream 102C may only include speech from Carol. The single-speaker audio streams 102A-C may be generated using a consistent time representation scheme (eg, consistent timestamps) that is consistent across the single-speaker audio streams 102A-C.

[0045] At operation 616, a plurality of single-speaker transcriptions are generated based on the plurality of single-speaker audio streams. Each single-speaker transcription in the plurality of single-speaker transcriptions may be generated based on a corresponding single-speaker audio stream in the plurality of single-speaker audio streams. Figure 1 As shown, the single-speaker audio streams 102A-C can be provided to the transcription component 160, and the transcription component 160 can generate single-speaker transcriptions 103A-C from the single-speaker audio streams 102A-C. Each single-speaker transcription 103A-C is generated from a corresponding single-speaker audio stream in the single-speaker audio streams 102A-C. Therefore, each single-speaker transcription 103A-C can be a transcription of speech only from the corresponding speaker, and not from the speech of other speakers. Specifically, the single-speaker transcription 103A can be a transcription of speech only from Alice, the single-speaker transcription 103B can be a transcription of speech only from Bob, and the single-speaker transcription 103C can be a transcription of speech only from Carol. Each of the single-speaker transcriptions 103A-C can be generated using a consistent time representation scheme of the underlying single-speaker audio streams. For example, each of the single-speaker transcriptions 103A-C can include an indication of the words spoken by the respective speaker and an indication of the time at which the words were spoken.

[0046] At operation 618, the multiple single-speaker transcriptions are merged into a multi-speaker output transcription. The multiple single-speaker transcriptions may be merged into a multi-speaker output transcription based on a consistent time representation scheme. The multi-speaker output transcription may correspond to multiple speakers. For example, Figure 1As shown, the transcription component 160 can merge the single-speaker transcriptions 103A-C into a merged transcription 104. The merged transcription 104 is a multi-speaker output transcription corresponding to the input audio stream 100. The single-speaker transcriptions 103A-C can be merged into the merged transcription 104 based on a consistent time representation scheme. For example, the merged transcription 104 can include an indication of the words spoken by each speaker (e.g., Alice, Bob, and Carol) in the input audio stream 100, an indication of the time at which the words were spoken, and an indication of the identity of the speaker who spoke the words. In some examples, the identity of the speaker can be determined based on the single-speaker transcriptions 103A-C from which the words were obtained. For example, it can be determined that the words obtained from the single-speaker transcription 103A were spoken by Ann, the words obtained from the single-speaker transcription 103B were spoken by Bob, and the words obtained from the single-speaker transcription 103C were spoken by Carol. Therefore, the identity of the first speaker of the multiple speakers who spoke the first portion of speech in the multi-speaker output transcription can be indicated in the multi-speaker output transcription. The identity of the first speaker may be indicated based on a first single-speaker transcription of the plurality of single-speaker transcriptions from which the first portion of speech was obtained.

[0047] Figure 7 is a flow chart illustrating an example of a cluster-based embedding vector derivation process that can be used according to the present disclosure. Operations 710-718 may be included in the cluster-based embedding vector derivation process. In some examples, Figure 7 Operations 710-718 may include the following after the cluster-based embedding vector derivation process is completed, such as the above reference Figure 3 As described above, when included in the completed cluster-based embedding vector derivation process, operations 710-718 can be performed once for the entire input audio stream after the input audio stream is fully generated. In some other examples, Figure 7 Operations 710-718 may be included in an ongoing cluster-based embedding vector derivation process, such as that described above with reference to Figure 4 As described above, when included in an ongoing cluster-based embedding vector derivation process, operations 710-718 may be performed in a streaming manner during the generation of the input audio stream, and may be repeatedly performed on different portions of the input audio stream as different portions of the input audio stream are generated. For example, operations 710-718 may be initially performed in Figure 4 The cluster-based EV derivation iteration 401 of FIG. 4 is performed (on the stream portion 400A), and then can be Figure 4 The cluster-based EV derivation iteration 401 is performed again (on the stream portion 400B).

[0048] At operation 710, at least a portion of an input audio stream is analyzed using a speaker enumeration model to estimate a time-varying number of concurrent speakers within the input audio stream at a plurality of time points. Figure 3 As shown, during the post-cluster-based embedding vector derivation, the speaker enumeration ML model 300 can analyze the entire input audio stream 100 to estimate the time-varying number of co-existing speakers in the input audio stream 100 at multiple time points. Specifically, the speaker enumeration ML model 300 can estimate the number of speakers speaking in the input audio stream 100 at any given time. As described above, for ongoing cluster-based EV derivation, different portions of the input audio stream (e.g., stream portion 400A, stream portion 400B, etc.) can be analyzed in a streaming manner.

[0049] At operation 712, a plurality of single-speaker segments of the input audio stream may be determined using the time-varying number of concurrent speakers. The plurality of single-speaker segments of the input audio stream are segments in which there is only one speaker. Figure 3 As shown, the time-varying number of concurrent speakers may be used to determine single speaker segments (STS) 311-316 of the input audio stream 100. Each single speaker segment (STS) 311-316 is a segment of the input audio stream 100 having only one speaker therein.

[0050] At operation 714, a plurality of segment embedding vectors are derived. One or more of the plurality of segment embedding vectors may be derived for each of the plurality of single-speaker segments. For example, Figure 3 As shown, embedding vectorization 310 may be performed on each STS 311-316 to derive a segment embedding vector (SEV) 301-308. Each STS 311-316 includes at least one of the SEVs 301-308. Figure 3In the example of , SEV 301 and 302 are derived for STS 311, segment embedding vector (SEV) 303 is derived for STS 312, SEV 304 and 305 are derived for STS 313, SEV 306 is derived for STS 314, SEV 307 is derived for STS 315, and SEV 308 is derived for STS 316. Each SEV 301-308 is an embedding vector that represents the audio characteristics of a person speaking in a corresponding STS in STS 311-316. For example, SEV 301 and 302 may represent the audio characteristics of a person speaking in STS 311. SEV 301 may be generated based on the speech of the first half of STS 311, and SEV 302 may be generated based on the speech of the second half of STS 312. Thus, even though they may represent the voice of the same person, SEV 301 and SEV 302 may differ slightly from one another because the characteristics of a person's voice (e.g., pitch, volume / amplitude, speaking speed, etc.) may vary slightly over time.

[0051] At operation 716, the plurality of segment embedding vectors are clustered into a plurality of embedding vector clusters. Figure 3 As shown, an embedding vector clustering algorithm 320 may be performed on SEVs 301-308 to cluster SEVs 301-308 into embedding vector clusters 330A, 330B, and 330C. The embedding vector clustering algorithm 320 may cluster SEVs 301-308 into embedding vector clusters 330A-C based on the relative similarities between SEVs 301-308. For example, SEVs 301, 302, and 306 clustered into embedding vector cluster 330A may have greater similarity to each other than to any of SEVs 303-305, 307, and 308 that are not included in embedding vector cluster 330A. Similarly, SEVs 304, 305, and 307 clustered into embedding vector cluster 330B may have greater similarity to each other than to any of SEVs 301-303, 306, and 308 that are not included in embedding vector cluster 330B. Furthermore, SEVs 303 and 308 clustered into embedding vector cluster 330C may have greater similarity to each other than to any of SEVs 301 - 302 and 304 - 307 that are not included in embedding vector cluster 330C.

[0052] Each embedding vector cluster 330A-C may correspond to a corresponding speaker of the plurality of speakers. Specifically, embedding vector cluster 330A corresponds to Ann, embedding vector cluster 330B corresponds to Bob, and embedding vector cluster 330C corresponds to Carol. In this example, since there are three speakers (Ann, Bob, and Carol) in the input audio stream 100, three embedding vector clusters 330A-C are formed. It should be noted that for cluster-based embedding vector derivation, the number of speakers in the input audio stream 100 (and the number of embedding vector clusters 330A-C formed by clustering) is not known in advance. Instead, the number of speakers and embedding vector clusters 330A-C are determined based on the results of clustering, such as based on the relative similarities between SEVs 301-308.

[0053] At operation 718, a plurality of representative embedding vectors are derived for the plurality of embedding vector clusters, wherein the plurality of speaker embedding vectors include the plurality of representative embedding vectors. Each representative embedding vector in the plurality of representative embedding vectors may be derived for a corresponding embedding vector cluster of the plurality of embedding vector clusters. For example, Figure 3 As shown, during representative embedding vector (REV) derivation 340, REV 331-333 may be derived from embedding vector clusters 330A-C. Specifically, REV 331A is derived from embedding vector cluster 330A, REV 331B is derived from embedding vector cluster 330B, and REV 331C is derived from embedding vector cluster 330C. REV 331A is an embedding vector representing SEVs 301, 302, and 306 in embedding vector cluster 330A. For example, REV 331A may be the centroid of SEVs 301, 302, and 306, or may be generated based on the centroid of SEVs 301, 302, and 306. REV 331B is an embedding vector representing SEVs 304, 305, and 307 in embedding vector cluster 330B. For example, REV 331B may be the centroid of SEVs 304, 305, and 307, or may be generated based on the centroids of SEVs 304, 305, and 307. REV 331C is an embedding vector representing SEVs 303 and 308 in embedding vector cluster 330C. For example, REV 331C may be the centroid of SEVs 303 and 308, or may be generated based on the centroids of SEVs 303 and 308. Then, REVs 331A-C may be used as TEVs 105A-C for respective speakers. Specifically, REV 331A may be used as TEV 105A for Ann, REV 331B may be used as TEV 105B for Bob, and REV 331C may be used as TEV 105C for Carol.

[0054] It is worth noting that Figure 7Operations 710 and 712 in relate to executing a speaker enumeration ML model (before embedding vectorization) to determine examples of single-speaker segments, and embedding vectorization is performed only on single-speaker segments. However, as described above, alternative techniques (e.g., those described above with reference to Figure 5 As mentioned above, in this alternative technique, the embedding vectors corresponding to the single-speaker segments are detected during the clustering process and then discarded, which means that they are not used to determine the representative embedding vector. Therefore, for the alternative technique, it is possible to skip Figure 7 To this end, Figure 7 In the example, dashed lines are used instead of solid lines to illustrate operations 710 and 712. In addition, for alternative techniques, segment embedding vectors may be determined for the entire input audio stream (or a portion thereof) instead of just for a single speaker segment. It is worth noting that Figure 5 The alternative techniques in can be used for both post-completion and ongoing cluster-based embedding vector derivation.

[0055] for Figure 5 Alternatively, the clustering performed at operation 716 may include discarding one or more of the plurality of segment embedding vectors corresponding to one or more multi-speaker segments of the input audio stream, the multi-speaker segments being portions of the input audio stream where multiple speakers are speaking simultaneously. Figure 5 As described, various techniques may be employed to detect that SEVs 504-506 correspond to portions of the input audio stream 500 where multiple speakers (e.g., Ann and Bob) are speaking simultaneously. In some examples, the embedding vector clustering algorithm 520 may examine the temporal (time-based) relationships between SEVs 501-509. Specifically, the embedding vector clustering algorithm 520 may examine neighborhoods of SEVs that include two or more consecutive adjacent SEVs (or that are otherwise temporally close to each other) to determine the amount of similarity or correlation between adjacent SEVs. For example, for a scenario where only one person speaks in adjacent SEVs, it may be expected that these adjacent SEVs have a high degree of similarity to one another because they correspond to the speech of the same person. In contrast, for a scenario where multiple speakers are speaking simultaneously in adjacent SEVs, it may be expected that these SEVs would not have a high degree of similarity because they correspond to the speech of more than one person. In Figure 5In the example of , it can be determined that adjacent SEVs 501-503 have a high degree of similarity to each other and therefore correspond to the same person (Ann). In addition, it can be determined that adjacent SEVs 507-509 also have a high degree of similarity to each other and therefore correspond to the same person (Bob). In contrast, it can be determined that adjacent SEVs 504-506 are not highly similar to each other and therefore correspond to portions of the input audio stream 500 where multiple speakers (Ann and Bob) are speaking simultaneously. For alternative techniques, these time-based analysis techniques can be used as Figure 7 as part of operation 716. Figure 5 As shown, a representative embedding vector (REV) 531A is determined for embedding vector cluster 530A and used as TEV 105A for Ann. In addition, REV 531B is determined for embedding vector cluster 530B and used as TEV 105B for Bob. However, it should be noted that REV is not determined for the discarded multi-speaker group 535.

[0056] An example system for transmitting and providing data will now be described in detail. Specifically, Figure 8 An example computing environment is shown in which embodiments described herein may be implemented. Figure 8is a diagram schematically illustrating an example of a data center 85, which can provide computing resources to users 70a and 70b (which can be referred to as users 70 or users 70 in the singular or plural) via user computers 72a and 72b (which can be referred to as computers 72 in the singular or computers 72 in the plural) through a communication network 73. The data center 85 can be configured to provide computing resources for executing applications permanently or on demand. The computing resources provided by the data center 85 may include various types of resources, such as gateway resources, load balancing resources, routing resources, network resources, computing resources, volatile and non-volatile memory resources, content delivery resources, data processing resources, data storage resources, data communication resources, etc. Each type of computing resource can be used in a variety of specific configurations. For example, data processing resources can be used as virtual machine instances, which can be configured to provide various Web services. In addition, resource combinations can be provided over a network and can be configured as one or more Web services. These instances can be configured to execute applications, including web services, such as application services, media services, database services, processing services, gateway services, storage services, routing services, security services, encryption services, load balancing services, application services, etc. These services can be configured by setting or customizing applications, and can be configured in size, execution, cost, latency, type, duration, accessibility, and any other dimension. These web services can be configured as available infrastructure for one or more clients, and can include one or more applications configured as software for a platform or one or more clients. These web services can be provided by one or more communication protocols. These communication protocols may include, for example, hypertext transfer protocol (HTTP) or non-HTTP protocols. These communication protocols may also include, for example, more reliable transport layer protocols, such as transmission control protocol (TCP), and less reliable transport layer protocols, such as user datagram protocol (UDP). Data storage resources may include file storage devices, block storage devices, etc.

[0057] Each type or configuration of computing resources can be of different sizes, such as large resources (consisting of many processors, large amounts of memory, and / or large storage capacity) and small resources (consisting of fewer processors, smaller amounts of memory, and / or smaller storage capacity). For example, a customer may choose to allocate multiple small processing resources as web servers and / or allocate one large processing resource as a database server.

[0058] The data center 85 may include servers 76a and 76b (which may be referred to herein as server 76 in the singular or as multiple servers 76 in the plural) that provide computing resources. These resources may be provided as bare metal resources or virtual machine instances 78a-b (which may be referred to herein as virtual machine instance 78 in the singular or as multiple virtual machine instances 78 in the plural). In this example, the resources also include audio stream processing virtual machines (ASPVMs) 79a-b, which are virtual machines configured to perform any or all of the above-mentioned multi-speaker audio stream separation, transcription, and diary techniques.

[0059] The availability of computing hardware virtualization technology provides benefits for providing large-scale computing resources to customers and allowing computing resources to be efficiently and securely shared among multiple customers. For example, virtualization technology can allow physical computing devices to be shared among multiple users by providing each user with one or more virtual machine instances hosted by the physical computing device. A virtual machine instance can be a software simulation of a specific physical computing system that acts as a different logical computing system. Such a virtual machine instance provides isolation between multiple operating systems that share a given physical computing resource. In addition, some virtualization technologies can provide virtual resources that span one or more physical resources, such as a single virtual machine instance with multiple virtual processors that span multiple different physical computing systems.

[0060] refer to Figure 8 , the communication network 73 may be, for example, a network of publicly accessible linked networks and may be operated by a variety of different parties, such as the Internet. In other embodiments, the communication network 73 may be a private network, such as a company or university network that is completely or partially inaccessible to non-privileged users. In still other embodiments, the communication network 73 may include one or more private networks that can access the Internet and / or be accessed from the Internet.

[0061] The communication network 73 may provide access to the computer 72. The user computer 72 may be a computer used by the user 70 or other customers of the data center 85. For example, the user computer 72a or 72b may be a server, a desktop or notebook personal computer, a tablet computer, a wireless phone, a personal digital assistant (PDA), an e-book reader, a game console, a set-top box, or any other computing device capable of accessing the data center 85. The user computer 72a or 72b may be directly connected to the Internet (e.g., via a cable modem or a digital subscriber line (DSL)). Although only two user computers 72a and 72b are depicted, it should be understood that multiple user computers may exist.

[0062] The user computer 72 may also be used to configure various aspects of the computing resources provided by the data center 85. In this regard, the data center 85 may provide a gateway or Web interface through which various aspects of its operation may be configured using a Web browser application executing on the user computer 72. Alternatively, a standalone application executing on the user computer 72 may access an application programming interface (API) exposed by the data center 85 to perform configuration operations. Other mechanisms for configuring the operation of the various Web services available on the data center 85 may also be used.

[0063] Figure 8 The server 76 shown in the figure can be a server appropriately configured to provide the above-mentioned computing resources, and can provide computing resources for executing one or more Web services and / or applications. In one embodiment, the computing resources can be virtual machine instances 78. In the example of virtual machine instances, each server 76 can be configured to execute an instance manager 80a or 80b (which can be referred to as an instance manager 80 or multiple instance managers 80 in the plural) capable of executing virtual machine instances 78. For example, the instance manager 80 can be a virtual machine monitor (VMM) or another type of program configured to allow virtual machine instances 78 to be executed on the server 76. As described above, each virtual machine instance 78 can be configured to execute all or part of an application program.

[0064] It should be understood that although the embodiments disclosed above discuss the context of virtual machine instances, other types of implementations can be utilized using the concepts and techniques disclosed herein. For example, the embodiments disclosed herein can also be used with computing systems that do not use virtual machine instances.

[0065] exist Figure 8 In the example data center 85 shown, a router 71 can be used to interconnect servers 76a and 76b. The router 71 can also be connected to a gateway 74, which is connected to a communication network 73. The router 71 can be connected to one or more load balancers, and can manage communications within the network in the data center 85 alone or in combination, for example, by appropriately forwarding packets or other data communications based on the characteristics of such communications (e.g., header information including source and / or destination addresses, protocol identifiers, sizes, processing requirements, etc.) and / or the characteristics of the dedicated network (e.g., routing based on network topology, etc.). It should be understood that for simplicity, various aspects of the computing systems and other devices of this example are shown without showing certain conventional details. In other embodiments, additional computing systems and other devices may be interconnected, and may be interconnected in different ways.

[0066] exist Figure 8In the illustrated example data center 85, the server manager 75 is also used to at least partially direct various communications to, from, and / or between servers 76a and 76b. Figure 8 A router 71 is depicted as being located between a gateway 74 and a server manager 75, but this is merely an exemplary configuration. In some cases, for example, a server manager 75 may be located between a gateway 74 and a router 71. In some cases, a server manager 75 may examine portions of incoming communications from a user computer 72 to determine one or more appropriate servers 76 to receive and / or process the incoming communications. The server manager 75 may determine the appropriate server to receive and / or process the incoming communications based on factors such as the identity, location, or other attributes associated with the user computer 72, the nature of the task associated with the communication, the priority of the task associated with the communication, the duration of the task associated with the communication, the size of the task associated with the communication, and / or estimated resource usage, as well as many other factors. The server manager 75 may, for example, collect or otherwise access state information and other information associated with various tasks, in order to, for example, assist in managing communications and other operations associated with such tasks.

[0067] It should be understood that Figure 8 The network topology shown in has been greatly simplified, and more networks and network devices can be used to interconnect the various computing systems disclosed herein. These network topologies and devices should be obvious to those skilled in the art.

[0068] It should also be understood that Figure 8 The data center 85 described in the foregoing is merely illustrative, and other implementations may be used. It should also be understood that the server, gateway, or other computing device may include any combination of hardware or software capable of interacting and performing the types of functions described, including but not limited to: desktop or other computers, database servers, network storage devices and other network devices, PDAs, tablet computers, cell phones, wireless phones, pagers, electronic organizers, Internet appliances, television-based systems (e.g., using set-top boxes and / or personal / digital video recorders), and various other consumer products that include appropriate communication capabilities.

[0069] In at least some embodiments, a server implementing part or all of one or more of the techniques described herein may include a computer system that includes or is configured to access one or more computer-accessible media. Fig. 9A computer system including or configured to access one or more computer-accessible media is depicted. In the illustrated embodiment, computing device 15 includes one or more processors 10a, 10b, and / or 10n (which may be referred to herein as "processor 10" in the singular or "processors 10" in the plural) coupled to system memory 20 via input / output (I / O) interface 30. Computing device 15 also includes a network interface 40 coupled to I / O interface 30.

[0070] In various embodiments, computing device 15 may be a uniprocessor system including one processor 10 or a multiprocessor system including a plurality of processors 10 (e.g., two, four, eight, or other suitable number). Processor 10 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 10 may be an embedded processor implementing any of a variety of instruction set architectures (ISAs), such as an x86, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each processor 10 may typically (but not necessarily) implement the same ISA.

[0071] System memory 20 may be configured to store instructions and data accessible to processor 10. In various embodiments, system memory 20 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / In the illustrated embodiment, program instructions and data that implement one or more desired functions (e.g., the methods, techniques, and data described above) are shown as stored within the system memory 20 as code 25 and data 26. Additionally, in this example, the system memory 20 includes audio stream processing instructions 27, which are instructions for performing any or all of the multi-speaker audio stream separation, transcription, and diaryization techniques described above.

[0072] In one embodiment, the I / O interface 30 may be configured to coordinate I / O traffic between the processor 10, the system memory 20, and any peripheral devices in the device (including the network interface 40 or other peripheral device interfaces). In some embodiments, the I / O interface 30 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., the system memory 20) into a format suitable for use by another component (e.g., the processor 10). In some embodiments, the I / O interface 30 may include support for devices attached via various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the functionality of the I / O interface 30 may be divided into two or more separate components, such as a north bridge and a south bridge. In addition, in some embodiments, some or all of the functionality of the I / O interface 30 (e.g., the interface with the system memory 20) may be directly incorporated into the processor 10.

[0073] The network interface 40 may be configured to allow data to be exchanged between the computing device 15 and other devices or devices 60 (e.g., other computer systems or devices) attached to the network or networks 50. In various embodiments, the network interface 40 may support communications via any suitable wired or wireless general purpose data network, such as, for example, an Ethernet type of network. Additionally, the network interface 40 may support communications via a telecommunications / telephone network (e.g., an analog voice network or a digital fiber optic communications network), via a storage area network (e.g., a Fibre Channel SAN (Storage Area Network)), or via any other suitable type of network and / or protocol.

[0074] In some embodiments, the system memory 20 may be an embodiment of a computer-accessible medium configured to store program instructions and data as described above for implementing embodiments of the corresponding methods and apparatus. However, in other embodiments, the program instructions and / or data may be received, sent, or stored on different types of computer-accessible media. In general, the computer-accessible medium may include a non-transitory storage medium or memory medium, such as a magnetic or optical medium, such as a disk or DVD / CD coupled to the computing device 15 via the I / O interface 30. The non-transitory computer-accessible storage medium may also include any volatile or non-volatile medium, such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM (read-only memory), etc., which may be included in some embodiments of the computing device 15 as the system memory 20 or another type of memory. In addition, the computer-accessible medium may include a transmission medium or signal, such as an electrical signal, an electromagnetic signal, or a digital signal transmitted through a communication medium (e.g., a network and / or a wireless link, such as those that can be implemented through the network interface 40).

[0075] A network established by an entity (e.g., a company or public sector organization) for providing one or more Web services (e.g., various types of cloud-based computing or storage) accessible via the Internet and / or other networks to a distributed group of clients may be referred to as a provider network. Such a provider network may include numerous data centers hosting various resource pools, such as collections of physical and / or virtualized computer servers, storage devices, network devices, etc., that are necessary to implement and distribute the infrastructure and Web services provided by the provider network. In some embodiments, resources may be provided to clients in various units related to the Web service, such as the amount of storage capacity for storage, processing power for processing, instances, sets of related services, etc. A virtual computing instance may, for example, include one or more servers with specified computing power (which may be specified by indicating the type and number of CPUs, the size of main memory, etc.) and a specified software stack (e.g., a specific version of an operating system, which in turn may run on a hypervisor).

[0076] Computing nodes (also referred to as operation nodes) can be implemented in various computing environments, such as commodity hardware computers, virtual machines, Web services, computing clusters, and computing tools. For convenience, any of these computing devices or environments can be described as a computing node.

[0077] In various embodiments, a variety of different types of computing devices may be used, alone or in combination, to implement the resources of the provider network, such as computer servers, storage devices, network devices, etc. In some embodiments, a client or user may be provided with direct access to a resource instance, for example, by providing the user with an administrator login and password. In other embodiments, the provider network operator may allow the client to specify execution requirements for a given client application and may run the application on behalf of the client on an execution platform suitable for the application (e.g., an application server instance, a Java JVM, etc.). TM The execution of an application is scheduled on a virtual machine (JVM), a general-purpose or special-purpose operating system, a platform that supports various interpreted or compiled programming languages ​​(such as Ruby, Perl, Python, C, C++, etc.), or a high-performance computing platform) without, for example, requiring the client to directly access an instance or execution platform. A given execution platform can utilize one or more resource instances in some implementations; in other implementations, multiple execution platforms can be mapped to a single resource instance.

[0078] In many environments, operators of provider networks that implement different types of virtualized computing, storage, and / or other network-accessible functions may allow customers to reserve or purchase access to resources in various resource acquisition models. Computing resource providers may provide customers with facilities to select and launch required computing resources, deploy application components to computing resources, and maintain applications executing in the environment. In addition, computing resource providers may provide customers with further facilities to quickly and easily increase or decrease the number and type of resources allocated to applications manually or through automatic expansion as the demand or capacity requirements of applications change. Computing resources provided by computing resource providers may be provided in the form of discrete units, which may be referred to as instances. An instance may represent a physical server hardware platform, a virtual machine instance executing on a server, or a combination of both. Instances of various types and configurations may be provided, including resources of different sizes executing different operating systems (OS) and / or hypervisors, as well as various installed software applications, runtimes, etc. Instances may also be provided in specific availability zones, for example, logical areas representing the underlying computing hardware, fault-tolerant regions, data centers, or other geographic locations. Instances may be replicated within or across availability zones to increase the redundancy of instances, and instances may be migrated within or across availability zones. As an example, the latency for a client to communicate with a particular server in an availability zone may be lower than the latency for a client to communicate with a different server. Therefore, instances can be migrated from a higher latency server to a lower latency server to improve the overall client experience.

[0079] In some embodiments, the provider network may be organized into multiple geographic regions, and each region may include one or more availability zones. Availability zones (which may also be referred to as availability containers) may include one or more different locations or data centers, which are configured in such a way that resources in a given availability zone can be isolated or isolated from failures in other availability zones. That is, it is expected that a failure in one availability zone may not cause a failure in any other availability zone. Therefore, the availability container of a resource instance is intended to be independent of the availability container of resource instances in different availability zones. Clients can protect their applications from failures in a single location by launching multiple application instances in each availability zone. At the same time, in some implementations, cheap and low-latency network connections can be provided between resource instances located in the same geographic region (and network transmissions between resources in the same availability zone may be faster).

[0080] As described above, a content provider may provide content to one or more clients. The term "content" used herein refers to any information that can be presented, and the term "content item" used herein refers to a collection of any such information that can be presented. For example, a content provider may provide one or more content provision services for providing content to a client. The content provision service may reside on one or more servers. The content provision service may be scalable to meet the needs of one or more clients, and its capabilities may be increased or decreased according to the number and type of incoming client requests. The portion of the content provision service may also be migrated to a location where the latency with the requesting client is reduced. For example, a content provider may determine the "edge" of a system or network associated with a content provision service, which is physically and / or logically closest to a specific client. The content provider may then, for example, "start", migrate resources, or otherwise use components associated with the determined edge to interact with a specific client. In some cases, this edge determination process may provide an effective technique for identifying and using components that are well suited for interacting with a specific client, and in some embodiments, may reduce the communication latency between a content provider and one or more clients.

[0081] In addition, some method or process blocks may be omitted in some implementations. The methods and processes described herein are not limited to any particular order, and the blocks or states associated therewith may be performed in other appropriate orders. For example, the blocks or states described may be performed in an order different from the order specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be performed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments.

[0082] It should also be understood that various items are shown as being stored in memory or on a storage device when in use, and these items or portions thereof may be transferred between memory and other storage devices for memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. In addition, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least partially in firmware and / or hardware, including but not limited to one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a computer-readable medium, such as a hard disk, memory, network, or portable media article, so as to be read by an appropriate drive or via an appropriate connection. Systems, modules, and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless and wire / cable-based media), and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital data packets or frames). In other embodiments, such computer program products may also take other forms. Therefore, the present invention may be practiced with other computer system configurations.

[0083] Furthermore, embodiments of the present disclosure may be described according to the following clauses:

[0084] Clause 1. A computing system comprising:

[0085] one or more processors; and

[0086] one or more memories having stored therein instructions that, when executed by one or more processors, cause the computing system to perform computing operations, the computing operations comprising:

[0087] deriving a plurality of speaker embedding vectors corresponding to a plurality of speakers in an input audio stream, wherein each speaker embedding vector in the plurality of speaker embedding vectors corresponds to a corresponding speaker in the plurality of speakers and represents a corresponding voice feature of the corresponding speaker;

[0088] executing a plurality of instances of a personalized noise suppression model on the input audio stream, wherein each instance of the plurality of instances of the personalized noise suppression model employs a corresponding speaker embedding vector from the plurality of speaker embedding vectors;

[0089] generating, by the plurality of instances of the personalized noise suppression model, a plurality of single-speaker audio streams corresponding to the input audio stream, wherein each of the plurality of single-speaker audio streams is generated by a corresponding instance of the plurality of instances of the personalized noise suppression model by outputting only sounds in the input audio stream corresponding to the corresponding speaker embedding vector, and wherein the plurality of single-speaker audio streams are generated using a consistent time representation scheme that is consistent among the plurality of single-speaker audio streams;

[0090] generating a plurality of single-speaker transcriptions based on the plurality of single-speaker audio streams, wherein each single-speaker transcription of the plurality of single-speaker transcriptions is generated based on a corresponding single-speaker audio stream of the plurality of single-speaker audio streams; and

[0091] The plurality of single-speaker transcriptions are merged into a multi-speaker output transcription corresponding to the plurality of speakers based on the consistent temporal representation scheme.

[0092] Clause 2. The computing system of clause 1, wherein deriving the plurality of speaker embedding vectors comprises performing a registration process in which the plurality of speakers are registered.

[0093] Clause 3. The computing system of clause 1, wherein deriving the plurality of speaker embedding vectors comprises performing cluster-based embedding vector derivation.

[0094] Clause 4. The computing system of clause 3, wherein the cluster-based embedding vector derivation is performed after the input audio stream is fully generated and is performed on the entirety of the input audio stream.

[0095] Clause 5. The computing system of clause 3, wherein the cluster-based embedding vector derivation is performed in a streaming manner during generation of the input audio stream and is repeatedly performed on different portions of the input audio stream as different portions of the input audio stream are generated.

[0096] Clause 6. A computer-implemented method comprising:

[0097] deriving a plurality of speaker embedding vectors corresponding to a plurality of speakers in an input audio stream, wherein each speaker embedding vector in the plurality of speaker embedding vectors corresponds to a corresponding speaker in the plurality of speakers and represents a corresponding voice feature of the corresponding speaker;

[0098] executing a plurality of instances of a personalized noise suppression model on the input audio stream, wherein each instance of the plurality of instances of the personalized noise suppression model employs a corresponding speaker embedding vector from the plurality of speaker embedding vectors;

[0099] generating, by the multiple instances of the personalized noise suppression model, a plurality of single-speaker audio streams corresponding to the input audio stream, wherein each of the multiple single-speaker audio streams is generated by a corresponding instance of the multiple instances of the personalized noise suppression model by outputting only sounds in the input audio stream corresponding to the corresponding speaker embedding vector;

[0100] generating a plurality of single-speaker transcriptions based on the plurality of single-speaker audio streams, wherein each single-speaker transcription of the plurality of single-speaker transcriptions is generated based on a corresponding single-speaker audio stream of the plurality of single-speaker audio streams; and

[0101] The plurality of single-speaker transcriptions are merged into a multi-speaker output transcription corresponding to the plurality of speakers.

[0102] Clause 7. The computer-implemented method of clause 6, wherein deriving the plurality of speaker embedding vectors comprises performing a registration process in which the plurality of speakers are registered.

[0103] Clause 8. The computer-implemented method of clause 6, wherein deriving the plurality of speaker embedding vectors comprises performing a cluster-based embedding vector derivation.

[0104] Clause 9. The computer-implemented method of clause 8, wherein performing the cluster-based embedding vector derivation comprises:

[0105] analyzing at least a portion of the input audio stream using a speaker enumeration model to estimate a time-varying number of concurrent speakers within the input audio stream at a plurality of time points;

[0106] determining a plurality of single-speaker segments having only one speaker in the input audio stream using the time-varying number of concurrent speakers;

[0107] deriving a plurality of segment embedding vectors, wherein one or more of the plurality of segment embedding vectors are derived for each single-speaker segment of the plurality of single-speaker segments;

[0108] clustering the plurality of segment embedding vectors into a plurality of embedding vector clusters; and

[0109] A plurality of representative embedding vectors are derived for the plurality of embedding vector clusters, wherein each representative embedding vector of the plurality of representative embedding vectors is derived for a corresponding embedding vector cluster of the plurality of embedding vector clusters, and wherein the plurality of speaker embedding vectors include the plurality of representative embedding vectors.

[0110] Clause 10. The computer-implemented method of clause 8, wherein performing the cluster-based embedding vector derivation comprises:

[0111] derive multiple segment embedding vectors;

[0112] clustering the plurality of segment embedding vectors into a plurality of embedding vector clusters, wherein the clustering comprises discarding one or more segment embedding vectors of the plurality of segment embedding vectors corresponding to one or more multi-speaker segments of the input audio stream; and

[0113] A plurality of representative embedding vectors are derived for the plurality of embedding vector clusters, wherein each representative embedding vector of the plurality of representative embedding vectors is derived for a corresponding embedding vector cluster of the plurality of embedding vector clusters, and wherein the plurality of speaker embedding vectors include the plurality of representative embedding vectors.

[0114] Clause 11. The computer-implemented method of clause 8, wherein the cluster-based embedding vector derivation is performed after the input audio stream is fully generated and is performed on the entirety of the input audio stream.

[0115] Clause 12. A computer-implemented method of clause 8, wherein the cluster-based embedding vector derivation is performed in a streaming manner during generation of the input audio stream and is repeatedly performed on different portions of the input audio stream as they are generated.

[0116] Clause 13. The computer-implemented method of Clause 6, wherein the plurality of single-speaker audio streams are generated using a consistent time representation scheme that is consistent among the plurality of single-speaker audio streams.

[0117] Clause 14. The computer-implemented method of clause 13, wherein merging the plurality of single-speaker transcriptions into a multi-speaker output transcription is performed based on the consistent time representation scheme.

[0118] Clause 15. The computer-implemented method of Clause 6, further comprising:

[0119] An identity of a first speaker of the plurality of speakers who spoke a first portion of speech in the multi-speaker output transcription is indicated in the multi-speaker output transcription, wherein the identity of the first speaker is indicated based on a first single-speaker transcription of the plurality of single-speaker transcriptions from which the first portion of speech was obtained.

[0120] Clause 16. One or more non-transitory computer-readable storage media having stored thereon computing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform computing operations comprising:

[0121] deriving a plurality of speaker embedding vectors corresponding to a plurality of speakers in an input audio stream, wherein each speaker embedding vector in the plurality of speaker embedding vectors corresponds to a corresponding speaker in the plurality of speakers and represents a corresponding voice feature of the corresponding speaker;

[0122] executing a plurality of instances of a personalized noise suppression model on the input audio stream, wherein each instance of the plurality of instances of the personalized noise suppression model employs a corresponding speaker embedding vector from the plurality of speaker embedding vectors;

[0123] generating, by the multiple instances of the personalized noise suppression model, a plurality of single-speaker audio streams corresponding to the input audio stream, wherein each of the multiple single-speaker audio streams is generated by a corresponding instance of the multiple instances of the personalized noise suppression model by outputting only sounds in the input audio stream corresponding to the corresponding speaker embedding vector;

[0124] generating a plurality of single-speaker transcriptions based on the plurality of single-speaker audio streams, wherein each single-speaker transcription of the plurality of single-speaker transcriptions is generated based on a corresponding single-speaker audio stream of the plurality of single-speaker audio streams; and

[0125] The plurality of single-speaker transcriptions are merged into a multi-speaker output transcription corresponding to the plurality of speakers.

[0126] Clause 17. The one or more non-transitory computer-readable storage media of clause 16, wherein deriving the plurality of speaker embedding vectors comprises performing cluster-based embedding vector derivation.

[0127] Clause 18. The one or more non-transitory computer-readable storage media of clause 17, wherein performing the cluster-based embedding vector derivation comprises:

[0128] analyzing at least a portion of the input audio stream using a speaker enumeration model to estimate a time-varying number of concurrent speakers within the input audio stream at a plurality of time points;

[0129] determining a plurality of single-speaker segments having only one speaker in the input audio stream using the time-varying number of concurrent speakers;

[0130] deriving a plurality of segment embedding vectors, wherein one or more of the plurality of segment embedding vectors are derived for each single-speaker segment of the plurality of single-speaker segments;

[0131] clustering the plurality of segment embedding vectors into a plurality of embedding vector clusters; and

[0132] A plurality of representative embedding vectors are derived for the plurality of embedding vector clusters, wherein each representative embedding vector of the plurality of representative embedding vectors is derived for a corresponding embedding vector cluster of the plurality of embedding vector clusters, and wherein the plurality of speaker embedding vectors include the plurality of representative embedding vectors.

[0133] Clause 19. One or more non-transitory computer-readable storage media of Clause 17, wherein the cluster-based embedding vector derivation is performed after the input audio stream is fully generated and is performed on the entire input audio stream.

[0134] Clause 20. One or more non-transitory computer-readable storage media of Clause 17, wherein the cluster-based embedding vector derivation is performed in a streaming manner during generation of the input audio stream and is repeatedly performed on different portions of the input audio stream as different portions of the input audio stream are generated.

[0135] Unless otherwise expressly stated or otherwise understood in the context in which it is used, conditional language used herein (e.g., "can," "might," "perhaps," "may," "for example," and other languages) is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not include these features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way necessary for one or more embodiments, or that one or more embodiments necessarily include logic (whether input or prompted by the author) for determining whether these features, elements, and / or steps are included or whether they are performed in any particular embodiment. The terms "include," "comprises," "has," and the like are synonymous and are used inclusively in an open-ended manner and do not exclude other elements, features, actions, operations, and the like. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so when used, for example, to connect a list of elements, the term "or" means one element, some elements, or all elements in the list.

[0136] Although certain example embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the inventions disclosed herein. Therefore, nothing in the foregoing description is intended to imply that any particular feature, characteristic, step, module, or block is necessary or indispensable. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions disclosed herein. The accompanying claims and their equivalents are intended to cover forms or modifications that fall within the scope and spirit of certain inventions disclosed herein.

Claims

1. A computing system comprising: one or more processors; as well as one or more memories, wherein the one or more memories store instructions that, when executed by one or more processors, cause the computing system to perform computing operations, the computing operations comprising: deriving a plurality of speaker embedding vectors corresponding to a plurality of speakers in an input audio stream, wherein each speaker embedding vector in the plurality of speaker embedding vectors corresponds to a corresponding speaker in the plurality of speakers and represents a corresponding voice feature of the corresponding speaker; executing a plurality of instances of a personalized noise suppression model on the input audio stream, wherein each instance of the plurality of instances of the personalized noise suppression model employs a corresponding speaker embedding vector from the plurality of speaker embedding vectors; generating, by the plurality of instances of the personalized noise suppression model, a plurality of single-speaker audio streams corresponding to the input audio stream, wherein each of the plurality of single-speaker audio streams is generated by a corresponding instance of the plurality of instances of the personalized noise suppression model by outputting only sounds corresponding to the corresponding speaker embedding vector from the input audio stream, and wherein the plurality of single-speaker audio streams are generated using a consistent time representation scheme that is consistent among the plurality of single-speaker audio streams; generating a plurality of single-speaker transcriptions based on the plurality of single-speaker audio streams, wherein each single-speaker transcription of the plurality of single-speaker transcriptions is generated based on a corresponding single-speaker audio stream of the plurality of single-speaker audio streams; and The plurality of single-speaker transcriptions are merged into a multi-speaker output transcription corresponding to the plurality of speakers based on the consistent temporal representation scheme.

2. The computing system of claim 1 , wherein deriving the plurality of speaker embedding vectors comprises performing a registration process in which the plurality of speakers are registered.

3. The computing system of claim 1, wherein deriving the plurality of speaker embedding vectors comprises performing a cluster-based embedding vector derivation.

4. The computing system of claim 3, wherein the cluster-based embedding vector derivation is performed after the input audio stream is fully generated and is performed on the entirety of the input audio stream.

5. The computing system of claim 3 , wherein the cluster-based embedding vector derivation is performed in a streaming manner during generation of the input audio stream, and the cluster-based embedding vector derivation is repeatedly performed on different portions of the input audio stream as they are generated.

6. A computer-implemented method comprising: deriving a plurality of speaker embedding vectors corresponding to a plurality of speakers in an input audio stream, wherein each speaker embedding vector in the plurality of speaker embedding vectors corresponds to a corresponding speaker in the plurality of speakers and represents a corresponding voice feature of the corresponding speaker; executing a plurality of instances of a personalized noise suppression model on the input audio stream, wherein each instance of the plurality of instances of the personalized noise suppression model employs a corresponding speaker embedding vector from the plurality of speaker embedding vectors; generating, by the plurality of instances of the personalized noise suppression model, a plurality of single-speaker audio streams corresponding to the input audio stream, wherein each of the plurality of single-speaker audio streams is generated by a corresponding instance of the plurality of instances of the personalized noise suppression model by outputting only sounds corresponding to the corresponding speaker embedding vector from the input audio stream; generating a plurality of single-speaker transcriptions based on the plurality of single-speaker audio streams, wherein each single-speaker transcription of the plurality of single-speaker transcriptions is generated based on a corresponding single-speaker audio stream of the plurality of single-speaker audio streams; as well as The plurality of single-speaker transcriptions are merged into a multi-speaker output transcription corresponding to the plurality of speakers.

7. The computer-implemented method of claim 6, wherein deriving the plurality of speaker embedding vectors comprises performing a registration process in which the plurality of speakers are registered.

8. The computer-implemented method of claim 6, wherein deriving the plurality of speaker embedding vectors comprises performing a cluster-based embedding vector derivation.

9. The computer-implemented method of claim 8, wherein performing the cluster-based embedding vector derivation comprises: analyzing at least a portion of the input audio stream using a speaker enumeration model to estimate a time-varying number of concurrent speakers within the input audio stream at a plurality of time points; determining a plurality of single-speaker segments having only one speaker in the input audio stream using the time-varying number of concurrent speakers; deriving a plurality of segment embedding vectors, wherein one or more of the plurality of segment embedding vectors are derived for each single-speaker segment of the plurality of single-speaker segments; clustering the plurality of segment embedding vectors into a plurality of embedding vector clusters; as well as A plurality of representative embedding vectors are derived for the plurality of embedding vector clusters, wherein each representative embedding vector of the plurality of representative embedding vectors is derived for a corresponding embedding vector cluster of the plurality of embedding vector clusters, and wherein the plurality of speaker embedding vectors include the plurality of representative embedding vectors.

10. The computer-implemented method of claim 8, wherein performing the cluster-based embedding vector derivation comprises: derive multiple segment embedding vectors; clustering the plurality of segment embedding vectors into a plurality of embedding vector clusters, wherein the clustering comprises discarding one or more segment embedding vectors of the plurality of segment embedding vectors corresponding to one or more multi-speaker segments of the input audio stream; as well as A plurality of representative embedding vectors are derived for the plurality of embedding vector clusters, wherein each representative embedding vector of the plurality of representative embedding vectors is derived for a corresponding embedding vector cluster of the plurality of embedding vector clusters, and wherein the plurality of speaker embedding vectors include the plurality of representative embedding vectors.

11. The computer-implemented method of claim 8, wherein the cluster-based embedding vector derivation is performed after the input audio stream is fully generated and is performed on the entirety of the input audio stream.

12. The computer-implemented method of claim 8, wherein the cluster-based embedding vector derivation is performed in a streaming manner during generation of the input audio stream, and the cluster-based embedding vector derivation is repeatedly performed on different portions of the input audio stream as they are generated.

13. The computer-implemented method of claim 6, wherein the plurality of single-speaker audio streams are generated using a consistent time representation scheme that is consistent across the plurality of single-speaker audio streams.

14. The computer-implemented method of claim 13, wherein merging the plurality of single-speaker transcriptions into a multi-speaker output transcription is performed based on the consistent time representation scheme.

15. The computer-implemented method of claim 6, further comprising: An identity of a first speaker of the plurality of speakers who spoke a first portion of speech in the multi-speaker output transcription is indicated in the multi-speaker output transcription, wherein the identity of the first speaker is indicated based on a first single-speaker transcription of the plurality of single-speaker transcriptions from which the first portion of speech was obtained.

Images