Systems and methods for speech processing using time-invariant speaker embeddings

Abstract

A speech processing system and method for multi-speaker conversation analysis are provided. The speech processing system includes a deep neural network trained for processing audio segments of an audio mixture of a multi-speaker conversation. The deep neural network includes a speaker-independent layer that generates speaker-independent outputs and a speaker bias layer that is applied individually once to each of the audio segments for each of multiple speakers in the audio mixture. The deep neural network also processes the time-invariant embeddings by inputting corresponding time-invariant speaker embeddings to assign each application of the speaker bias layer to a corresponding speaker individually. In this manner, the deep neural network generates data indicative of time-frequency activity regions for each of the multiple speakers in the audio mixture from the combination of the speaker bias outputs.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to speech processing, and more particularly to speaker separation and identification tasks in speech processing. [Background technology]

[0002] Traditional source separation systems for extracting target audio signals typically aim to separate only specific types of sounds, such as speech enhancement or instrument demixing. The targets are determined by a training scheme and may not change during testing. Traditional source separation approaches typically separate an audio mixture into only specific types of sources (e.g., separating vocals from background noise) or all sources in a mixture without using discriminants (e.g., separating individual speakers in a conference room), and then use post-processing to find the target signals. Recently, criteria-based approaches have emerged as a promising alternative, allowing auxiliary inputs such as class labels to indicate the desired sources. However, the available sets of criteria are typically mutually exclusive and lack flexibility.

[0003] In modern contexts, speech separation systems, such as those required for conference transcription, involve not only separating speakers but also transcribing the conference recording into machine-readable text and enhancing the transcription with speaker details. Such information is known as diarization information, which indicates who spoke when in the conference audio mixture. Traditionally, diarization is the process of dividing an audio stream into distinct segments according to speaker identities. Thus, speaker diarization is a combination of speaker segmentation and speaker clustering, where speaker segmentation deals with identifying speaker transitions in an audio stream and speaker clustering deals with grouping speech segments based on speaker characteristics.

[0004] However, providing accurate diarization information and accurate conference transcription is a challenging task, especially due to the dynamics of multi-speaker conversational speech. Multi-speaker conversational speech involves multiple speakers speaking intermittently, interspersed with segments of inactivity between single and multiple speakers. In particular, overlapping speech, where two or more people speak simultaneously, is known to pose significant challenges not only for speech processing tasks such as automatic speech recognition (ASR), but also for diarization. Furthermore, two key tasks in multi-speaker speech processing—voice separation / enhancement and diarization—must be performed accurately, either sequentially or simultaneously, significantly affecting the overall quality of the speech processing system. However, the choice of whether to start processing with diarization or voice separation / enhancement is not obvious, making designing such a speech processing system extremely challenging.

[0005] With the advent of artificial intelligence (AI), deep learning, and machine learning (ML), challenging tasks are increasingly being performed using AI / ML-based systems. One popular deep learning technique used in diverse applications is the neural network. Neural networks have been used in numerous applications across various fields over the past few decades due to their ability to reproduce and model nonlinear processes. Neural networks can learn (or train) by processing examples, each containing a known "input" and "outcome," and forming probability-weighted associations between the two, which are stored within the network's own data structure. Training a neural network from a given example is typically performed by finding the difference between the network's processed output (often a prediction) and a target output, also referred to herein as the training label. This difference represents the error that training seeks to reduce. The network then uses this error value to adjust the weighted associations according to a learning rule. By repeating this adjustment, the neural network produces outputs that are increasingly similar to the target output. After a sufficient number of these adjustments have been made, training can be terminated based on certain criteria.

[0006] This type of training is typically called supervised learning. In supervised learning, a neural network "learns" to perform a task by considering examples, typically without being programmed with task-specific rules. For example, in image recognition, a neural network might learn to identify images that contain cats by analyzing examples of images labeled "cat" or "not cat," and then using those results to identify cats in other images. The neural network does this without any prior knowledge of cats, such as fur, tails, whiskers, and cat-like faces. Instead, it automatically generates discriminative characteristics from the examples it processes.

[0007] However, such supervised learning requires labeling images as cats or dogs. Such labeling is a tedious and laborious process. Also, in this image recognition example, the labeling is unambiguous: images include cats, dogs, or neither cats nor dogs. Such unambiguous labeling is not always possible. For example, some training applications tackle sequence problems with variable timing. The time variable can create one-to-many or many-to-one ambiguities in training, where the input sequence is of a different length than the output sequence. Audio processing systems dealing with speech transcription and speaker diarization are examples of such systems, which must accurately process long, unsegmented speech / audio inputs to produce accurate and augmented transcription results.

[0008] However, due to the different types and nature of tasks involved in such speech processing, different neural networks have traditionally been used to perform different tasks, and increasing the number of neural networks means that the overall speech processing system becomes more complex, which is undesirable.

[0009] Therefore, there is a need for an advanced system that overcomes the above-mentioned drawbacks. To that end, there is a need for a technical solution that overcomes the above-mentioned challenges. More specifically, there is a need for a system that provides enhanced speech separation and speaker identification over conventional speech processing systems. Summary of the Invention

[0010] Simplifying device structures and processes by removing elements and / or operational steps while achieving the same results is believed to be beneficial to various technical fields. These advantages are evident in mechanical and civil engineering and manufacturing processes, but are also beneficial to other technical fields such as ML and AI.

[0011] For example, many AI applications use a cascaded execution of a sequence of neural networks to achieve a desired result. Each neural network in the sequence serves a specific purpose, allowing all neural networks to jointly achieve the desired result. There are many reasons that justify the cooperation of multiple neural networks. Some of these reasons relate to the specifics of the neural network's execution during online testing or the specifics of offline training of the neural network. For example, in a speech processing system designed for multi-speaker conversation analysis, the two separate tasks of speaker separation and speaker diarization can be performed using two different neural networks, each trained separately for the task of interest. However, this increases the overall complexity and computational requirements of the speech processing system due to the need to expend separate effort to train each of the two neural networks. Furthermore, at runtime, each neural network execution requires separate memory and computing resources.

[0012] Some embodiments are based on the recognition that in some AI applications, it is advantageous to reduce the number of neural networks used in a cascade execution, even though this may potentially increase the complexity of training the neural networks, because the available computing power during execution may be less than the computing power available during the training phase, and this is further due to the statistical nature of the execution and training of neural networks.

[0013] Specifically, even if there is unlimited labeled training data (which is rare), training may not provide a trained neural network that performs the task with the desired accuracy. This is because the training data for training a neural network is different from the input data processed by the trained neural network, and machine learning must rely on, and hopes, but does not guarantee, that the statistical distribution of the features used to process the training data will be similar to the statistical distribution of the input data. Thus, statistical uncertainty is introduced by each neural network in a cascaded execution of a sequence of neural networks.

[0014] With this understanding in mind, an objective of some embodiments is to improve the accuracy of speech processing systems configured for multi-speaker conversation analysis of long, unsegmented speech recordings for applications such as conference transcription. The embodiments are based on the understanding that it is possible to perform this task by implementing multiple neural networks. For example, a first neural network followed by a second speech separation neural network informed by the output of the first neural network can be used to perform conditional diarization of speakers speaking in a recorded audio mixture given the initial-letter-only signatures of the speakers' speech.

[0015] However, some embodiments are based on the recognition that it is possible to reduce the number of neural networks in the above-mentioned cascade implementation by training neural networks capable of processing audio mixtures formed by the speech of multiple speakers given the speakers' voice signatures and extracting individual speaker recordings from the mixture. Specifically, some embodiments are based on the recognition that it is advantageous to jointly perform diarization and separation on input audio mixtures formed by the speech of multiple speakers because the complexity of designing and training multiple neural networks is reduced. Furthermore, performing such joint diarization and separation with a single neural network jointly trained for both tasks can reduce the overall computational requirements of a speech processing system designed for multi-speaker conversation analysis and transcription.

[0016] Some embodiments are based on the recognition that both speaker diarization and speaker separation tasks are highly interdependent, and therefore jointly addressing both is highly beneficial. To that end, the subtasks to be solved in either of these two are similar. That is, diarization is the task of determining the active speaker in each time frame, while mask-based source extraction for speaker separation identifies the dominant active speaker for each time-frequency (TF) bin. The only difference between the two is the time-to-time-frequency resolution.

[0017] Some embodiments recognize that traditional joint diarization and source separation systems are based on spatial mixture models using time-varying mixture weights. Thus, estimates of the prior probabilities of mixture components after appropriate quantization provide diarization information about who speaks and when, while the posterior probabilities have TF resolution and can be used to extract each source present in the mixture by masking or beamforming. However, a challenge in these traditional systems is the initialization of the mixture weights or posterior probabilities. In some known systems, the initialization of the time-varying mixture weights is achieved by guided source separation (GSS), which uses manual annotation of segment boundaries and their subsequent estimates. Furthermore, some prior solutions use initialization schemes that exploit the specifications of the conference data, where only one speaker is active most of the time. Therefore, clustering short time segments leads to sufficiently distinguishable clusters, from which initial values ​​for the parameters of the spatial mixture model can be established.

[0018] However, all of these previous solutions to the joint diarization and separation problem are computationally intensive and depend on the availability of multi-channel inputs, which is more preferably handled by offline algorithms that are less suitable for online or real-time processing of conferences of any length.

[0019] Some embodiments are based on the realization that the computational complexity and demands of jointly performing diarization and separation can be reduced by a single neural network, such as a deep neural network, trained for both tasks simultaneously.

[0020] Some embodiments are further based on the recognition that constrained diarization can be used to directly perform speech extraction, thereby eliminating the need for additional dedicated separation or extraction systems. As is well known, constrained diarization only has time resolution for estimating the activity of each speaker, while speech extraction is typically performed using time-frequency resolution estimation (so-called masks). Therefore, some embodiments disclose a deep neural network whose output is extended by the frequency dimension, and the entire speech processing system based on this deep neural network can be directly used for diarization and speaker separation without the need for an additional dedicated separation network. Processing in this manner eliminates the need for an additional neural network, which simplifies the computational and memory requirements for multi-speaker speech separation applications.

[0021] Some embodiments are based on the recognition that the task of target speaker separation in speech processing is processing time-varying audio signals based on time-varying enrollment information for each speaker in a multi-speaker speech scenario. Examples of such processing include neural networks with attention architectures. However, some embodiments are based on the recognition that time-varying enrollment information can have poor performance and be computationally inefficient, especially when the enrollment information is recorded during a long meeting that includes regions of speaker inactivity.

[0022] To address this shortcoming, in some embodiments, an architecture that has proven effective for diarization tasks is modified such that the modified system can perform speaker separation by extending the architecture to output data indicative of the time-frequency activity regions of each of multiple speakers.

[0023] As a result, some embodiments disclose a deep neural network that includes a speaker-independent layer that is applied to an audio mixture of multiple speakers and generates an output common to all of the multiple speakers, and a speaker bias layer that is applied repeatedly and individually to each of the multiple speakers. Each application of the speaker bias layer is assigned individually to a corresponding speaker. The speaker bias layer receives two types of inputs: a common input for each application of the speaker bias layer generated by the speaker-independent layer, and an individual input of a speaker embedding that indicates the audio signature of the corresponding speaker. In this way, speaker embeddings can be estimated for each speaker and used in multiple runs of the deep neural network that process different segments of the audio mixture.

[0024] Thus, some embodiments disclose a method for processing an audio mixture formed by one or a combination of simultaneous and continuous speech from multiple speakers. The method uses a processor coupled with stored instructions implementing the method, which, when executed by the processor, perform the method steps. The method steps include receiving an audio mixture and identifying information in the form of a time-invariant speaker embedding for each of the multiple speakers. The audio mixture is processed with a deep neural network, the deep neural network including: (1) a speaker-independent layer applied to the audio mixture of the multiple speakers to generate a speaker-independent output common to all of the multiple speakers; and (2) a speaker bias layer applied independently once for each of the multiple speakers to the speaker-independent output to generate a speaker-biased output for each of the multiple speakers. Each application of the speaker bias layer is individually assigned to a corresponding speaker by inputting a corresponding time-invariant speaker embedding. The deep neural network further extracts data indicative of time-frequency activity regions for each of the multiple speakers in the audio mixture from the combination of the speaker bias outputs. The extracted data is then appropriately rendered.

[0025] For example, the extracted data may be rendered as a time-frequency mask and combined with the audio mixture to generate an output for a single target speaker, which in some embodiments includes a text output showing the speech transcription data for the single speaker.

[0026] In some embodiments, the deep neural network is trained using a weakly supervised training process, in which training data is in the form of temporal annotation data and includes at least ground truth diarization-related data and ground truth separated source data. Furthermore, during the training, a diarization loss is calculated based on the weak labels, a separation loss is calculated based on the strong labels, and the deep neural network is trained using a loss obtained by combining the diarization loss and the separation loss. In some embodiments, the training is joint training.

[0027] In some embodiments, the deep neural network further comprises a joint estimation layer configured to extract data indicative of time-frequency activity regions for each of the multiple speakers.

[0028] Thus, in some embodiments, a speech processing system is disclosed, the speech processing system comprising a memory storing instructions and a processor configured to execute the stored instructions to implement a method, the method including receiving an audio mixture formed by one or a combination of simultaneous and continuous speech of multiple speakers and identification information in the form of a time-invariant speaker embedding for each of the multiple speakers. The audio mixture is processed with a deep neural network, the deep neural network including (1) a speaker-independent layer applied to the audio mixture of the multiple speakers to generate a speaker-independent output common to all of the multiple speakers, and (2) a speaker bias layer applied independently once for each of the multiple speakers to the speaker-independent output to generate a speaker bias output for each of the multiple speakers, each application of the speaker bias layer being individually assigned to a corresponding speaker by inputting a corresponding time-invariant speaker embedding. The method further includes extracting data indicative of time-frequency activity regions for each of the multiple speakers in the audio mixture from the combination of speaker bias outputs. The extracted data is then rendered.

[0029] Other features and advantages will become more readily apparent from the following detailed description when considered in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0030] [Figure 1A] 1 is a block diagram illustrating an audio processing system for processing an audio mixture according to one embodiment of the present disclosure. FIG. [Figure 1B] FIG. 1 is a block diagram illustrating the operation of an audio processing system for processing an audio mixture according to one embodiment of the present disclosure. [Figure 2] FIG. 1 is a detailed block diagram of an audio processing system according to an embodiment of the present disclosure. [Figure 3A] FIG. 1 illustrates an example architecture of a deep neural network for a speech processing system according to an embodiment of the present disclosure. [Figure 3B]FIG. 1 illustrates an example of training a deep neural network using a weakly supervised training strategy, according to one embodiment of the present disclosure. [Figure 3C] FIG. 10 illustrates another example of the operation of a deep neural network at different training and inference stages, according to an embodiment of the present disclosure. [Figure 3D] FIG. 1 is a schematic diagram illustrating inner layers of a deep neural network, according to some embodiments of the present disclosure. [Figure 4] FIG. 1 is a flow diagram illustrating a method for performing audio processing according to one embodiment of the present disclosure. [Figure 5A] FIG. 1 is a block diagram illustrating an example use of a voice processing system according to an embodiment of the present disclosure. [Figure 5B] FIG. 10 is a block diagram illustrating another example use of a voice processing system according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a block diagram illustrating an audio processing system for processing an audio mixture, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0031] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form to avoid obscuring the disclosure. Various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0032] As used herein and in the claims, when used in conjunction with a list of one or more components or other items, the terms “for example,” “for instance,” and “such as,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, are each to be construed as open-ended, meaning that such a list should not be viewed as excluding other additional components or items. The term “based on” means based at least in part on. Furthermore, it is to be understood that the phraseology and terminology used herein are for purposes of description and should not be viewed as limiting. Any headings used herein are for convenience only and do not have any legal or restrictive effect.

[0033] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Additionally, the same reference numbers and names in the various drawings refer to the same elements. System Overview

[0034] 1A is a block diagram 100a illustrating a voice processing system 104 for processing an audio mixture 102. The audio mixture 102 may correspond to a multi-speaker conversation including long, unsegmented speech from multiple speakers. The audio mixture 102 is thus formed by one or a combination of simultaneous and continuous speech from multiple speakers. For example, the audio mixture 102 may be captured during a conference involving multiple participants, each participant corresponding to a speaker. The audio mixture 102 may be captured via a single microphone or multiple microphones, which may then be combined to generate the audio mixture 102, which is received at an input interface (not shown in FIG. 1A) of the voice processing system 104.

[0035] The audio processing system 104 may be embodied as part of audio processing software, an audio processing system, a standalone speaker device, a conferencing application, a videoconferencing system, etc. The audio processing system 104 may also be embodied as part of a remote software platform, a remote server, a computing system, etc., that can receive and process the audio mixture 102 via a communication channel. The audio processing system 104 may be used to generate an output 106 corresponding to separated and augmented audio signals corresponding to each of multiple speakers in the audio mixture 102. For example, the output 106 may correspond to a conference transcript generated along with the identity of each of multiple speakers who are part of the conference conversation. In another example, the output 106 corresponds to a text output showing speech transcription data for one speaker. For example, the text output may be in the form of written text detailing the words spoken by each speaker.

[0036] To that end, the speech processing system 104, embodied in any of the above forms, includes a memory for storing instructions and the like, and a processor (shown later in FIG. 2). The memory may be a non-transitory computer-readable medium that stores data in the form of instructions or computer programs. The stored instructions are executable by the processor to achieve one or more desired objectives of the speech processing system 104. For example, one objective may be to implement a method for speech processing, including receiving input in the form of an audio mixture 102 containing one or a combination of simultaneous and continuous speech from multiple speakers, and identification information 108 for each of the multiple speakers.

[0037] The identification information 108 for each of the multiple speakers may be in the form of a time-invariant speaker embedding (hereinafter, the identification information 108 and the time-invariant speaker embedding 108 are used interchangeably to mean the same thing), where each speaker embedding represents the speech signature of the corresponding speaker. The time-invariant speaker embedding 108 includes a vector of speech signatures of the multiple speakers without the time dimension. To that end, the time-invariant speaker embedding 108 may represent a digital representation of the speech of each of the multiple speakers, a bias signal associated with each of the multiple speakers, a vector of speech profiles for each of the multiple speakers, a speaker embedding vector, etc.

[0038] The speech processing system 104 further comprises a deep neural network 110. The deep neural network 110 includes multiple layers, including an input layer, one or more hidden layers, and an output layer.

[0039] In some embodiments, the input audio mixture 102 is split into a sequence of audio segments at the input of the deep neural network 110. Each split audio segment is then processed by the deep neural network 110.

[0040] The deep neural network 110 includes speaker-independent layers that are applied to an audio mixture or a sequence of audio segments of the audio mixture 102 to generate a speaker-independent output 110a that is common to all of the multiple speakers.

[0041] The deep neural network 110 further includes a speaker bias layer applied to the speaker-independent output, applied once independently to each of the multiple speakers, thus generating a speaker-biased output for each of the multiple speakers, each application of the speaker bias layer being individually assigned to a corresponding speaker by inputting a corresponding time-invariant speaker embedding 108.

[0042] In an embodiment, a speaker bias layer is applied to each audio segment for each of multiple speakers, and each application of the speaker bias layer is assigned individually to the corresponding speaker. Furthermore, the time-invariant speaker embedding 108 is shared between processing of different audio segments.

[0043] To that end, the speaker bias layer generates a combination of speaker bias outputs 110b, which is a combination of the speaker bias outputs for each of the multiple speakers.

[0044] Further, based on the combination of the speaker bias outputs, data 110c indicative of time-frequency activity regions for each of the multiple speakers is extracted by the deep neural network 110 and then appropriately rendered to generate the desired output 106. The output 106 corresponds to separated and extended audio signals corresponding to each of the multiple speakers of the audio mixture 102. In one example, the extracted data 110c includes a time-frequency mask including estimates of time-frequency activity regions for each of the multiple speakers subjected to a nonlinear function. The time-frequency mask is combined with the audio mixture 102, and based on the combination, an output 106 for a single speaker from the multiple speakers is generated. For example, the output 106 may be in the form of a signal corresponding to the speech of each of the multiple speakers or a text output indicative of speech transcription data for each of the multiple speakers.

[0045] In this manner, the deep neural network 110 is configured as a single neural network to perform the diarization task of identifying time-based activity regions for each speaker and the separation task of identifying time-frequency-based activity regions for each speaker. This is typically done by extending intermediate outputs of the deep neural network 110, such as the outputs generated by a speaker bias layer, in the frequency dimension. This extension of the intermediate outputs of the deep neural network 110 includes generating a speaker bias output 110b as a conditional diarization output for each of multiple frequencies (actually, each of multiple speakers) that are part of the input audio mixture 102. Thus, outputting this conditional diarization output for each frequency produces speaker separation information.

[0046] To that end, the deep neural network 110 further comprises a joint estimation layer for extracting data indicative of the time-frequency activity regions of each of the multiple speakers in each audio segment.

[0047] 1B is a block diagram 100b illustrating the operation of an audio processing system 104 including a deep neural network 110, according to one embodiment of the present disclosure. Block diagram 100b shows that diarization network 111 can generate a time-resolution output 111a for audio mixture 102. This output 111a includes speaker activity domain information for speakers k=1 and k=2, etc., at different time instances T, e.g., time instance t1, time instance t2, etc. Thus, output 111a is a "who spoke when" diarization output, but does not provide information about each speaker's activity within each frequency, which is required to identify and consequently separate the speakers.

[0048] However, by using a deep neural network 110, the operation of this diarization network 111 is transformed (109) to provide output 111b in the time-frequency dimension. Essentially, the deep neural network 110 provides output 111b containing speaker activity domain information for different speakers, such as speakers k=1 and k=2, at different frequencies F at different time instances T, e.g., time instance t1, time instance t2, etc. As a result, output 111b contains output in the time-frequency TF domain, providing sufficient information to identify the speaker at each frequency. It also provides the times and frequencies at which a particular speaker was active, as well as the degree to which that speaker was active in the output, thereby providing a wealth of speaker separation and identification information.

[0049] This is advantageous over systems based on the cascading use of a diarization network 111 and multiple components, such as neural networks, that perform multi-speaker conversation analysis on long, unsegmented audio recordings for applications such as conference transcription, where the audio mixture 102 is input. Identifying speakers in long, unsegmented recordings or speech is the task of a diarization module, performed by one neural network, and then separating each speaker's recording is the task of a separate audio separation module, which may traditionally be based on a separate neural network. In this task, the interaction between the diarization module and the audio separation module is important. One possible cascade relies on overlap-aware diarization followed by an informed separation module (e.g., target speaker extraction). However, this cascade is suboptimal, and combining them into a single module can be beneficial. Therefore, the speech processing system 104 of FIG. 1A overcomes this drawback by using a single neural network, the deep neural network 110, to perform joint conditional diarization and separation and generate the output 106 based on this architecture. This output 106 is then extracted and rendered for different uses, one of which is to generate an augmented meeting transcript. For example, the extracted data 110c may be rendered at output 106 to a speech recognition system for further enhancement, which in turn may be used to generate a speaker-attributed transcript at output 106. Using speaker identification information computed from the output of the preliminary diarization system, diarization can be performed by a deep neural network 110 that is "informed" or "conditioned" on the identities of the different speakers in the audio mixture 102.

[0050] Extracting data 110c indicative of joint conditional diarization and separation information for multiple speakers using the deep neural network 110 employed by the speech processing system 104 is advantageous because it reduces the complexity of using separate neural networks or cascades of neural networks for separate tasks. Furthermore, the deep neural network 110 includes a neural network pre-trained for a classification-based objective before being fine-tuned for signal reconstruction. The classification-based objective simply attempts to classify whether each of the K speakers present in the recording is actively speaking or not in each time slot. This uses a binary cross-entropy loss between the estimated speaker activity and the ground truth speaker activity in each time slot. Training with a classification loss does not separate speech signals from multiple speakers speaking simultaneously. Once this loss converges, the speaker activation layer is copied F times to initialize F output layers capable of estimating time-frequency masks. The time-frequency masks are capable of separating overlapping speech and are trained for signal reconstruction. Examples of signal reconstruction objectives include the mean absolute error, mean squared error, or signal-to-distortion ratio between the estimated speech signal and the ground truth speech signal for each speaker. Training a deep neural network 110 using only the signal reconstruction objective results in poorer performance (sometimes not converging at all) compared to pre-training on a classification-based objective before fine-tuning on the signal reconstruction objective. This training schedule allows the trained neural network parameters to be initialized first on an easier classification task before moving on to the more difficult signal reconstruction task, which increases the likelihood that the network optimization will get stuck in a suboptimal local minimum.

[0051] Typically, diarization only has time resolution for each speaker's activity estimate, while speech extraction is typically performed using time-frequency resolution estimates (so-called masks). As a result, a single deep neural network 110 performing a joint conditional diarization and separation task can be extended by the frequency dimension, allowing the speech processing system 104 to be used directly for diarization and speaker separation without an additional dedicated separation network. Doing so allows the speech processing system 104 to be used more efficiently, as it eliminates the need for an additional neural network and simplifies the computational and memory requirements for multi-speaker speech separation applications.

[0052] 2 is a detailed block diagram illustrating an audio processing system 104 according to one embodiment of the present disclosure. The audio processing system 104 includes a processor 112 and a memory 114. The memory 114 is configured to store instructions for audio processing. The operations of the audio processing system 104 may be embodied in the form of instructions stored in the memory 114 and executed by the processor 112.

[0053] In some embodiments, memory 114 is configured to store deep neural network 110 to facilitate speech processing. Memory 114 corresponds to at least one of RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other storage media usable to store desired information and accessible by speech processing system 104. Memory 114 includes non-transitory computer storage media in the form of volatile and / or non-volatile memory. Memory 114 may be removable, non-removable, or a combination thereof. Exemplary memory devices include solid-state memory, hard drives, optical disk drives, etc.

[0054] The memory 114 stores instructions for causing the speech processing system 104 to perform method steps for enabling joint diarization and separation processing of an input audio mixture 102 formed by one or a combination of simultaneous and consecutive speech by multiple speakers. To that end, the processor 112 includes a receiving module 116 configured to receive the input audio mixture 102. The receiving module 116 also receives the identification information 108 described above in FIG. 1A. To that end, the receiving module 116 is configured to receive signals from one or more transmitting devices, such as one or more microphones transmitting audio signals corresponding to the speech utterances of different speakers. The receiving module 116 communicates with one or more external devices or systems, or internal modules, to exchange information or control signals. To that end, the receiving module 116 is configured to communicate with the transmitting devices or other compatible devices wirelessly or via a wired connection. The receiving module 116 can also store and update received signal data in the memory 114 unit for subsequent retrieval and analysis. The receiver module 116 receives power from an external power source or via an internal power source and may be integrated into a larger system or used as a stand-alone unit.

[0055] The receiving module 116 then submits the received audio mixture 102 and identification information 108 to a segmentation module 116a. The segmentation module 116a is configured to segment the audio mixture 102 into a sequence of audio segments. For example, the audio mixture 102 is transformed into the STFT domain with a 64 ms window and a 16 ms shift. In the STFT domain, a logarithmic spectrogram and mel frequency cepstral coefficients (MFCCs) are stacked as input features for the deep neural network 110.

[0056] In an embodiment, the segmentation module 116a is configured to split a long unsegmented audio mixture into smaller segments, such as during training of the deep neural network 110. For example, a 10-minute recording of a meeting is split into chunks of 1-minute recording and then sent to the deep neural network 110.

[0057] In another example, during post-processing, the segmentation module 116a is configured to segment long periods of activity in the audio mixture 102 at silence positions according to speaker activity estimates, such that no segment exceeds 12 seconds. In this example, a minimum segment length of 40 frames (0.64 seconds) is used, which is only necessary if overestimation is not used.

[0058] Each segmented audio segment is then processed by a deep neural network 110. The deep neural network 110 includes at least a speaker-independent layer 202 and a speaker-bias layer 204. In addition, the deep neural network also includes a joint estimation layer 206.

[0059] The speaker-independent layer 202 is applied to the audio mixture 102 or to the sequence of audio segments generated by the segmentation module 116a to generate a speaker-independent output 202a common to all of the multiple speakers (equivalent to the speaker-independent output 110a shown in FIG. 1A). The speaker bias layer 204 is applied independently once to the speaker-independent output 202a for each of the multiple speakers. Each application of the speaker bias layer 204 is individually assigned to the corresponding speaker through the use of a time-invariant speaker embedding 108. The speaker bias layer 204 generates a speaker bias output 204a (equivalent to the speaker bias output 110b shown in FIG. 1A) for each speaker of the multiple speakers.

[0060] In the case of segmentation, the deep neural network 110 is run once per audio segment to generate a speaker-independent output 202a, and the speaker bias layer 204 is applied independently once per speaker to each audio segment to generate a speaker-biased output 204a with each speaker's corresponding assignment to the speaker-independent output 202a. During execution of the deep neural network 110, the time-invariant speaker embedding 108 is shared between processing of different audio segments of the sequence of segmented audio segments of the audio mixture 102.

[0061] The time-invariant speaker embedding 108 is combined with the speaker-independent outputs 202a for each speaker of the multiple speakers to generate a combination of speaker bias outputs 204a. The combination of speaker bias outputs 204a is then used to extract data indicative of the time-frequency activity regions for each speaker of the multiple speakers. To this end, the combination of speaker bias outputs 204a is sent to the combined estimation layer 206 of the deep neural network 110.

[0062] The joint estimation layer 206 uses a combination of the speaker bias outputs 204a to provide an extraction of data indicative of the time-frequency activity regions for each speaker in each of the multiple speakers of the audio mixture 102 or audio segments of the audio mixture 102. The extracted data is provided to the extraction module 118. The extraction module 118 further processes the extracted data to provide a time-frequency mask including an estimate of the time-frequency activity region for each speaker of the multiple speakers, to which a nonlinear function is applied. For example, the nonlinear function may include a sigmoid operation, a softmax operation, a threshold operation, a smoothing operation, a morphological operation, etc.

[0063] Furthermore, the extraction module 118 may be used to combine the extracted time-frequency masks with the audio mixture 102 to generate an output 106 for each target speaker from the multiple speakers. This output 106 is then rendered in one or more output modalities, including, but not limited to, text, speech, audio, video, audio-video, multimedia, or a combination thereof. For example, the text modality may include speech transcription data for each of the multiple speakers along with the speaker's corresponding ID. The output 106 generated by the deep neural network 110 is highly accurate and of high quality, providing reliable speech transcription data, especially for online meeting transcription applications. Furthermore, the deep neural network 110 architecture described herein is an extension of the diarization neural network architecture and is more efficient, computationally feasible, and requires less computational power to implement than neural network architectures that include a cascade of neural networks (each neural network performing a different function).

[0064] The operation of the different layers of deep neural network 110 is explained in more detail below in the description of FIG. 3A.

[0065] 3A illustrates an example architecture 300a of the deep neural network 110 of the speech processing system 104. The architecture 300a is used to implement the functionality of the deep neural network 110 for performing joint conditional diarization and speaker separation, as previously described in FIGS. 1A and 2, according to one embodiment of the present disclosure.

[0066] The input audio mixture 102 is provided to a deep neural network 110, which first performs a Short Time Fourier Transform (STFT) operation 102a on the audio mixture 102 to transform the audio mixture 102 into the time-frequency domain as a spectrogram, which is an acoustic time-frequency domain representation of the audio mixture 102.

[0067] In an example of segmentation, the input audio mixture 102 is segmented into a sequence of audio segments, such as by a segmentation module 116a, and these segments are then transformed into the time-frequency domain as a spectrogram, which is the acoustic time-frequency domain of the audio mixture 102, by a short-time Fourier transform (STFT) operation 102a performed by the deep neural network 110.

[0068] Generally, a spectrogram includes elements defined by values ​​such as pixel values ​​in the time-frequency domain. Each value of each element is identified by its coordinate in the time-frequency domain. For example, time periods in the time-frequency domain are represented as columns, and frequency bands in the time-frequency domain are represented as rows.

[0069] The audio signal 102 is associated with the speech of one or more speakers. In a conference setting, the one or more speakers are conference participants. As part of the speaker separation task, the deep neural network 110 is trained to identify each of the one or more speakers and to identify each speaker's pronunciation. Thus, the deep neural network 110 is trained jointly on two tasks: (1) separating the received speech mixture 102 into one or more segments, each corresponding to a speaker among the multiple speakers, and (2) mapping corresponding identification information for each speaker to a corresponding separated segment, such that each separated segment is defined by the corresponding speaker producing the utterance associated with that segment. The deep neural network 110 is configured to operate sequentially on different tasks in the processing pipeline of the speech processing system 104.

[0070] First, the deep neural network 110 receives the audio mixture 102, which may be divided into a sequence of audio segments and converted into a time-frequency domain representation 102b by an STFT operation 102a in a speaker-independent layer 302 of the deep neural network 110 (the speaker-independent layer 302 is equivalent to the speaker-independent layer 202). The time-frequency domain representation 102b is in the form of an STFT of the input or received audio mixture 102. The STFT operation 102a is performed, for example, with a window of 64 milliseconds and a shift of 16 milliseconds. Since the time resolution is represented by T and the frequency resolution is represented by F, the time-frequency domain representation 102b of the input audio mixture 102 has dimensions T×F. To that end, the speaker-independent layer 302 is configured to process the input audio mixture 102 into speaker-independent features based on the time-frequency characteristics of the input audio mixture 102. For example, one approach uses features from the STFT-based time-frequency representation 102b to process the input audio mixture 102 into speaker-independent features 302a of dimension T×Z1, where Z1 corresponds to a feature dimension referred to as the speaker-independent feature dimension. In one embodiment, the speaker-independent layer 302 generates speaker-independent features 302a in the form of embedding vectors. An example of a feature input to the speaker-independent layer 302 in the STFT domain is a logarithmic spectrogram and mel-frequency cepstral coefficients (MFCCs) stacked together. The speaker-independent features 302a output by the speaker-independent layer 302 form a speaker-independent output common to all of the multiple speakers (hereinafter, the terms speaker-independent features 302a and speaker-independent output 302a may be used interchangeably to mean the same thing). The speaker-independent output 302a is equivalent to the speaker-independent output 110a and the speaker-independent output 202a.

[0071] Additionally, the first concatenation layer 308 is configured to concatenate or combine the speaker-independent output 302 a with time-invariant speaker embeddings 108 corresponding to the identities 108 associated with the multiple speakers. The purpose of the concatenation layer 308 is to combine the speaker-independent output 302 a with a separate input of a speaker embedding from the time-invariant speaker embeddings 108, independently and once for each of the multiple speakers, for each application of the speaker bias layer 304 assigned to one of the multiple speakers. Each speaker embedding is indicative of the phonetic signature of the corresponding speaker, and the time-invariant speaker embeddings 108 are shared between processing of different audio segments of the audio mixture 102, and the time-invariant speaker embeddings are constant across executions of the deep neural network 110.

[0072] In one example, the time-invariant speaker embedding 108 includes speaker embeddings for multiple speakers and has dimensions K×E, where K represents the number of speakers and E represents the dimension of the embedding. The first concatenation layer 308 generates a first concatenated output 308a with dimensions K×T×(Z1+E). Here, the speaker-independent features 302a with speaker-independent feature dimension Z1 are copied K times, and each copy is concatenated with one of K speaker embeddings with embedding dimension E. The embedding vectors of the time-invariant speaker embedding 108 serve as condition vectors for multiple speakers, and as a result of the concatenation, are concatenated to every frame of the audio mixture 102 or every audio segment of a sequence of audio segments in the audio mixture 102. To this end, in a conference scenario, assuming that the total number of speakers in the conference is known and embedding vectors representing the speakers are available, the deep neural network 110 is configured to simultaneously estimate the activities of all speakers. This combined estimation ultimately exhibits high diarization performance.

[0073] Rather than relying on enrollment utterances, the speaker-independent layer 302 of the deep neural network 110 estimates a speaker profile from the region of a presumed single speaker in the recording to be diarized. Furthermore, this type of processing does not require precise knowledge of the number of speakers, as long as it is possible to determine the maximum number of potentially present speakers. One method for determining this maximum number of speakers is attention-based speaker identification.

[0074] The first concatenated output 308a is then passed to the speaker bias layer 304 of the deep neural network 110, which generates a combination of speaker bias outputs 304a of dimension K×T×Z2, where Z2 is the feature dimension output of the speaker bias layer 304, referred to as the bias output feature dimension Z2.

[0075] The speaker bias layer 304 is applied once for each segment of the concatenated output 308a corresponding to each audio segment in the sequence of audio segments of the audio mixture 102, independently for each speaker of the multiple speakers. Thus, the speaker bias layer 304 operates independently on the portion of the first concatenated output 308a corresponding to each speaker in turn, thereby effectively processing an input of dimension T×(Z1+E) independently K times and outputting an output of dimension T×Z2, where Z2 represents the bias output feature dimension. The output of the speaker bias layer 304 is a combination of the speaker bias outputs 304a of dimension T×Z2 generated K times. The K outputs processed separately by the speaker bias layer 304 can be considered together to form a combination of speaker bias outputs 304a. The combination of speaker bias outputs 304a is equivalent to the speaker bias output 110b and the speaker bias output 204a.

[0076] Both dimensions Z1 and Z2 are hyperparameters that essentially determine the size of each layer in the deep neural network 110: the speaker-independent layer 302 for Z1 and the speaker-bias layer 304 for Z2. In the context of neural networks, hyperparameters are parameters that are set before the learning process begins and determine the architecture and behavior of the network. Unlike the weights and biases of a neural network, which are learned during training, hyperparameters are predefined by the user and remain constant throughout the training process.

[0077] Furthermore, the second concatenation layer 310 is configured to concatenate, transpose, and rearrange the combination of speaker bias outputs 304a to generate a second concatenated output 310a of dimension T×(KZ2).

[0078]

number

[0079] The joint estimation layer 306 is the last layer of the deep neural network 110, which is extended by replicating it F times in the frequency dimension, where F is the number of frequency bins. This modification enables the deep neural network 110 to generate a time-frequency mask. Therefore, the deep neural network 110 outputs data representing a separation output 306b. Optionally, the deep neural network 110 may output data representing a diarization output 306a. Alternatively, the separation output 306b may be further processed to obtain a diarization output. In this way, the deep neural network 110 can simultaneously perform joint conditional diarization and separation based on the received audio mixture 102 and the time-invariant speaker embedding 108. Example Implementation

[0080]

number

[0081]

number

[0082] The deep neural network 110 generates an embedding I that represents the speaker. k Given a mixture Y t,f From target speaker k's speech activity A t,k It is trained to predict speaker embedding I k is estimated from the recording rather than using the enrollment utterance. k is equivalent to the time-invariant speaker embedding 108 and is time-invariant since it does not have a time dimension. Furthermore, a connection layer of the deep neural network 110, such as the connection estimation layer 306, simultaneously estimates the activities of all speakers within a segment of the audio mixture 102 or for the audio mixture 102 itself, given the time-invariant embedding 108.

[0083]

number

[0084]

number

[0085] As shown in FIG. 3A, the deep neural network 110 is composed of three components: a speaker-independent layer 302, a speaker-bias layer 304, and a joint estimation layer 306, with a stacking operation performed between them.

[0086]

number

[0087]

number

[0088]

number

[0089]

number

[0090] Therefore, the deep neural network 110 can perform the separation at the output layer by providing the extraction of the mask given in equation (5) above.

[0091] Next, training of the deep neural network 110 will be described.

[0092]

number

[0093] This is followed by a second training phase in which the deep neural network 110 is initialized with the parameters of the pre-training phase, i.e., all learnable weights and bias parameters from the neural network blocks of Figure 3A (speaker-independent block 302, speaker-bias block 304, and joint-estimation block 308) are further optimized during this second training phase to estimate the time-frequency activity domain, as opposed to the time-activity domain used in the first phase.

[0094] The training of the deep neural network 110 may be direct training, multi-objective sequential training, multi-objective joint training, weakly supervised training, or any combination thereof.

[0095] Multi-objective sequential training involves using a deep neural network (DNN) core structure with different output layers that are used sequentially in different training stages, where in a first training stage the DNN core structure is trained with a first output layer that outputs diarization information of the audio mixture 102, and in a second training stage the first output layer is replaced with a second output layer that outputs extracted data.

[0096] Direct training involves supervised learning of labeled speech utterances of individual speakers in the audio mixture 102 .

[0097] Multi-objective simultaneous training includes using a multi-head neural network architecture with different heads to generate different outputs of extracted data, the different outputs corresponding to at least data indicative of speech pronunciations of each speaker forming the audio mixture and corresponding identification information associated with the speech pronunciations.

[0098] Weakly supervised training involves training a deep neural network 110 based on temporal annotation data associated with the audio mixture 102. Existing speech separation systems typically rely on simulated data for training neural networks to perform speaker separation tasks. However, using a weakly supervised training methodology, the deep neural network 110 may be trained with real data rather than simulated data, with the temporal annotations of the real data of the audio mixture serving as weakly supervised labels. Weakly supervised training is further described in conjunction with FIG. 3B.

[0099] FIG. 3B is a diagram 300b illustrating an example of training a deep neural network 110 using a weakly supervised training strategy, according to one embodiment of the present disclosure.

[0100] As is well known in machine learning and neural networks, supervised training involves training a neural network based on labeled data. Labels indicate ground truth values ​​or classes of the training data. In supervised learning, a large labeled dataset is essential. However, this is not always possible. Therefore, FIG. 3B illustrates an example of weakly supervised training, which involves training a deep neural network 110 based on temporal annotation data associated with an audio mixture 102. The temporal annotation data consists of labels specifying the times when each of the k speakers in the mixture is active and inactive, but the term weakly labeled refers to the lack of separated speech signals in areas of speaker overlap. Temporal annotation data can be obtained from real data by listening to a recording of the audio mixture 102 and labeling when speakers are active. Obtaining separated speech signals for each speaker (especially during overlap areas) is impossible with real data, and fully supervised training can only be performed using simulated mixture signals.

[0101] Weakly supervised training involves training a neural network using training data with only partial annotations or labels. For example, as shown in diagram 300b, training audio signal samples 312 are obtained from a weakly labeled dataset containing real audio mixture data 314 associated with weak labels 322, or are generated by creating an artificial audio mixture from separated speech signals. If the training audio signal samples 312 are associated with strong labels 324, the corresponding weak labels 322 can also be obtained from the strong labels 324. The real (audio mixture) data 314 may contain periods in which multiple speakers are speaking simultaneously, and it is not possible to obtain separate individual signals from each speaker. However, it is possible to obtain the time regions in which each speaker in the audio mixture is actively speaking. These annotated speaker activity regions from the real data 314 correspond to the weak labels 322 in FIG. 3B. With the real data 314, only the diarization loss 318 can be calculated; the separation loss 320 is not used. For simulated data 316, audio mixture data is artificially generated from separated single-speaker speech signals to obtain separated speech signals for each speaker, which corresponds to strong labels 324. In this case, strong labels mean that the time-frequency activity regions for each speaker are known, as opposed to only the time activity regions in the case of weak labels. For simulated data, separation loss 320 can be used for training purposes, although a combination of diarization loss 318 and separation loss 320 may also be used for each audio signal in the simulated dataset.

[0102] Typically, speech separation systems rely on simulated data to train neural networks that perform the task of speaker separation. However, using a weakly supervised training methodology, the deep neural network 110 may be trained on real data 314 instead of or in addition to simulated data 316, where temporal annotations of the training audio signal samples 312 for the real data 314 serve as weakly supervised labels 322. The temporal annotation data includes ground truth data, including data on diarization information and data on separated sound sources, derived based on ground truth estimates of all speaker activities.

[0103] To that end, the deep neural network 110 is jointly trained to calculate a diarization loss 318 based on weak labels 322 of the training audio signal samples 312 and a separation loss 320 based on strong labels of the training audio signal samples 312. Furthermore, the diarization loss 318 and the separation loss 320 are combined to determine a total loss 326 for the deep neural network 110.

[0104] Thus, the deep neural network 110 is trained in two stages: a pre-training stage and a training stage. In the pre-training stage, the deep neural network 110 is trained to distinguish between multiple speakers. At the end of this pre-training stage, the last layer of the deep neural network 110, the combined estimation layer 306, is copied F times to achieve the desired time-frequency resolution.

[0105] This is followed by a second training phase in which the deep neural network 110 is initialized with parameters from the pre-training phase, including the weights and bias values ​​for each layer of the deep neural network 110 learned in the pre-training phase.

[0106] The pre-training stage uses the sum of the diarization losses between the estimated activities and the ground truth activities for all speakers. The diarization loss 318 is, in one example, the binary cross entropy (BCE) loss.

[0107] In the training phase, a separation loss 320 is used, which in one example is a signal reconstruction loss, since it implicitly takes into account phase information.

[0108] The training of the deep neural network 110 using pre-training and training stages is further illustrated in FIG. 3C.

[0109] FIG. 3C is a schematic diagram 300c illustrating another example of the operation of the deep neural network 110 at different training stages, according to one embodiment of the present disclosure.

[0110]

number

[0111]

number

[0112]

number

[0113] where τ is a threshold and δ(x) represents the Kronecker delta, which evaluates to 1 if x is true and 0 otherwise. If the dilation window is larger than the erosion window, speech activity will be overestimated. At activity transitions, the speaker's signal is cut off, resulting in segments of constant speaker activity. When the deep neural network 110 is trained to fill gaps of short inactivity (e.g., pauses between words), median-based smoothing may be used. However, when the deep neural network 110 is trained to predict time-frequency reconstruction masks for each speaker, mask values ​​are typically smaller than speaker activity estimates, short inactivity produces zeros, and values ​​in overlapping regions are smaller than values ​​in non-overlapping regions. Thresholding and closing morphological operations help fill the gaps.

[0114]

number

[0115] where b is the segment index and T b is the set of frame indices belonging to segment b, and k b is the index of the speaker active in segment b to be extracted, and ε=0.0001 is a small value introduced for stability.

[0116]

number

[0117]

number

[0118]

number

[0119] During training in Figure 3C, the ground truth speaker activity A at each time slot t t,k The labels are known because they are used in the calculation of the diarization loss 318. Therefore, the time-invariant speaker embedding vectors 108 containing the identities of each of the k speakers can be derived from these ground truth activity labels. However, during testing, the ground truth speaker activity A t,k This information is not available and must be estimated from the initial diarization information 108a, which is the purpose of switching between training and test points in Figure 3C.

[0120] FIG. 3D is a schematic diagram 300d illustrating the internal layers of a deep neural network 110 according to some embodiments of the present disclosure. The deep neural network 110 can be a network or circuit of artificial neural networks composed of artificial neurons or nodes. Thus, the deep neural network 110 is an artificial neural network used to solve artificial intelligence (AI) problems. The connections of biological neurons are modeled as weights between nodes in an artificial neural network. Positive weights reflect excitatory connections, while negative weight values ​​represent inhibitory connections. All inputs 334 of the deep neural network 110 may be modified by the weights and summed. Such activity is called a linear combination. Finally, an activation function controls the amplitude of the output 336 of the neural network 106. For example, the acceptable range of the output 336 is typically between 0 and 1, or between -1 and 1. Artificial networks can be used for predictive modeling, adaptive control, and other applications where they may be trained via a training dataset. Self-learning arising from experience occurs within the network, allowing it to draw conclusions from complex and seemingly unrelated sets of information.

[0121] The internal structure shown in Figure 3D may be used to implement different layers of deep neural network 110, as previously described, to perform the speech processing functions of speech processing system 104. Figure 4 is a flow diagram illustrating a method 400 for performing speech processing according to one embodiment of the present disclosure.

[0122] Method 400 is triggered at 402 upon detection of an input at one or more microphones of a speech processing system, such as speech processing system 104. Following the triggering, an audio mixture and identification information are received as input. As described in FIG. 1A, audio mixture 102 includes consecutive and simultaneous speech from multiple speakers, and speech processing system 104 receives this audio mixture 102 at step 404. The speech processing system further receives identification information 108 for each of the multiple speakers of audio mixture 102 in the form of a time-invariant speaker embedding. As previously described, identification information 108 includes, by way of example, a digital representation of the speech footprint or speech characteristics of each of the multiple speakers. Furthermore, as described in FIG. 3A, identification information 108 in the form of a time-invariant speaker embedding is estimated according to Equation (4) by using segments of audio mixture 102 in which only one speaker is active. This time-invariant speaker embedding is held constant throughout execution of the deep neural network. Also, when the audio mixture 102 is split into a sequence of audio segments, such as by the splitting module 116a, the time-invariant speaker embeddings are shared between processing of different audio segments of the sequence of audio segments.

[0123] In step 404, the audio mixture 102, or each audio segment of the divided audio mixture, is processed by a deep neural network. For example, as shown in Figures 2, 3A, 3B, 3C, and 3D, a deep neural network 110 is used to process the audio mixture 102. As shown in Figure 3A, the deep neural network 110 includes a speaker-independent layer 302 that is applied to the audio mixture 102 after transformation into the STFT domain to generate a speaker-independent output 302a common to all of the multiple speakers. In addition, the deep neural network 110 includes a speaker bias layer 304 that is applied independently once to the speaker-independent output 302a to generate a speaker-biased output for each of the multiple speakers. Each application of the speaker bias layer is individually assigned to a corresponding speaker by inputting the time-invariant speaker embedding 108 corresponding to that application (however, this time-invariant speaker embedding remains the same across multiple runs of the deep neural network 110). As a result, each application of the speaker bias layer 304 is individually assigned to a corresponding speaker, producing a combination of speaker bias outputs 304a. In the case of segmenting the audio mixture 102, the deep neural network 110 is run once for each of the audio segments by using the speaker-independent layer 302 to generate a speaker-independent output 302a from the input audio mixture 102 and combining the speaker-independent output 302a with an individual input of the time-invariant speaker embedding 108 representing the phonetic signature of the corresponding speaker for each application of the speaker bias layer 304 assigned to one of the speakers. Here, the time-invariant speaker embedding 108 is shared between processing of different audio segments of the audio mixture 104.

[0124]

number

[0125] 3C, at test time, the deep neural network 110 provides at output 106 an estimate of the signal for speaker k by multiplying the mask obtained in equation (5) by the STFT of the input audio at test or evaluation time, such as the received audio mixture 102. When multi-channel input is available, an estimated mask is utilized to compute the beamformer coefficients as an alternative to mask multiplication for source extraction.

[0126] Finally, the extracted data is rendered appropriately at 410. For example, the extracted data, e.g., after being combined with a multi-speaker audio mixture signal, e.g., in the form of a mask, provides activity data for the target speaker, which is used to render a speech transcription for the target speaker.

[0127] In this manner, the deep neural network 110 is used to implement the method 400 in the audio processing system 104 to provide speech transcription-related data for long audio recordings using online processing of the pre-trained deep neural network 110.

[0128] FIG. 5A is a block diagram 500a illustrating an example use of the voice processing system 104 according to one embodiment of the present disclosure.

[0129] Block diagram 500a illustrates a conference scenario in which multiple speakers, e.g., speaker S1, speaker S2, and speaker S3, are conversing in an online conference via computing device 502. The conversation corresponds to a multi-speaker conversation that generates an audio mixture, such as audio mixture 102. Audio mixture 102 is received by speech processing system 104. Speech processing system 104 may be embodied as part of computing device 502 or may be communicatively coupled to computing device 502. Speech processing system 104, including the aforementioned deep neural network 110, processes audio mixture 102 and generates output 106 at an output interface. Output 106 also includes a conference transcript for each of the multiple speakers (speaker S1, speaker S2, and speaker S3) along with identification of which speaker has which transcript.

[0130] For example, computing device 502 includes a display on which conference transcripts for all speakers are displayed, e.g., conference transcript 106a corresponds to speaker S1 at a first time instance, conference transcript 106b corresponds to speaker S2 at a second time instance, conference transcript 106c corresponds to speaker S3 at a third time instance, conference transcript 106n corresponds to speaker S1 at the nth time instance, etc.

[0131] Thus, the speech processing system 104 can generate an augmented audio transcript for a multi-speaker conversation of speakers (speaker S1, speaker S2, and speaker S3) in real time using a single deep neural network 110.

[0132] FIG. 5B is a block diagram 500b illustrating an example use of the audio processing system 104 according to one embodiment of the present disclosure.

[0133] Block diagram 500b illustrates a conferencing scenario in which multiple speakers, e.g., speaker S1 with corresponding audio signal 504a, speaker S2 with corresponding audio signal 504b, and speaker S3 with corresponding audio signal 504c, are conversing with a single microphone 504e, such as in a conference room. For example, microphone 504e may be associated with a teleconferencing device or a videoconferencing device. Furthermore, more than one microphone may be present in the environment represented by block diagram 500b, but their illustration has been omitted for the sake of brevity of this disclosure.

[0134] Thus, microphone 504e provides an audio mixture 504f (equivalent to audio mixture 102) formed by the simultaneous and / or sequential speech of multiple speakers (speaker S1, speaker S2, and speaker S3). Audio mixture 504f is then received by speech processing system 104, including deep neural network 110, as described in the previous embodiments. Speech processing system 104 may be embodied as part of computing device 502 or communicatively coupled to computing device 502. Speech processing system 104 may be a standalone computing device. In other examples, speech processing system 104 may be an application in the form of computer-executable instructions executed by a processor of a computing device to provide the functionality described in all the above embodiments. The speech processing system 104 includes the aforementioned deep neural network 110, which processes the audio mixture 504f and produces, at an output interface, an output 106 including a conference transcript for each of a plurality of speakers (speaker S1, speaker S2, and speaker S3), along with identification of which speaker has which transcript.

[0135] For example, meeting transcript 106x corresponds to speaker S1 at a first time instance, meeting transcript 106y corresponds to speaker S2 at a second time instance, meeting transcript 106z corresponds to speaker S3 at a third time instance, and so on.

[0136] Thus, the speech processing system 104 can use a single deep neural network 110 to generate an augmented audio transcript for a multi-speaker conversation of speakers (speaker S1, speaker S2, and speaker S3) in real time.

[0137] 6 is a block diagram 600 illustrating an audio processing system 104 for processing an audio mixture 102, according to some embodiments of the present disclosure. In some example embodiments, the block diagram 600 includes one or more microphones 606 that collect data including the audio mixture 102 from the environment 602.

[0138] The audio processing system 104 includes a hardware processor 608. The hardware processor 608 is in communication with a computer storage memory, such as a memory 610. The memory 610 contains stored data, including algorithms, instructions, and other data implemented by the hardware processor 608. Depending on the requirements of a particular application, it is envisioned that the hardware processor 608 includes two or more hardware processors. The two or more hardware processors may be either internal or external hardware processors. The audio processing system 104 may incorporate other components, including output interfaces and transceivers, among other devices.

[0139] In some alternative embodiments, the hardware processor 608 is connected to a network 604 that is in communication with the audio sources that generate the audio mixture 102. The network 604 includes, by way of non-limiting example and not limitation, one or more local area networks (LANs) and / or wide area networks (WANs). The network 604 also includes an enterprise-wide computer network, an intranet, and the Internet. The audio processing system 104 includes one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources includes one or more devices that cooperate in the distributed environment of the network 604.

[0140] In some other alternative embodiments, the hardware processor 608 is connected to a network-enabled server 614, which is connected to the client device 616. The network-enabled server 614 corresponds to a dedicated computer connected to a network that runs software intended to process client requests received from the client device 616 and provide appropriate responses on the client device 616. The hardware processor 608 is connected to an external memory device 618 that stores all necessary data used in the target audio signal extraction, and to a transmitter 620. The transmitter 620 facilitates the transmission of data between the network-enabled server 614 and the client device 616. Additionally, an output 622 associated with the target audio signal and localization information of the target audio signal is generated.

[0141] The audio mixture 102 is further processed by a deep neural network 110. The deep neural network 110 is trained using the audio mixture 102 and the identities 108 of the multiple speakers.

[0142] The deep neural network 110 processes the audio mixture 102 and the identification information 108 and generates, at output, data indicative of conditional diarization information, which includes information about the utterances of each of the multiple speakers, and speaker separation information, which includes information about the corresponding ID of each of the multiple speakers. Thus, the simplified architecture of the deep neural network 110 is configured to perform both speaker separation and speaker diarization tasks without relying on enrollment data of the multiple speakers.

[0143] Many modifications and other embodiments of the disclosures described herein will suggest themselves to one skilled in the art to which these disclosures pertain having the benefit of the teachings presented in the foregoing description and the associated drawings. It is to be understood that the disclosure is not limited to the particular embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, while the foregoing description and the associated drawings describe example embodiments in the context of example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, combinations of elements and / or functions other than those expressly described above are contemplated, for example, as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. 1. A method for processing an audio mixture formed by one or a combination of simultaneous and continuous speech of multiple speakers, the method using a processor coupled with stored instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, the method comprising: receiving an audio mixture and identification information in the form of a time-invariant speaker embedding for each of the plurality of speakers; and processing the audio mixture with a deep neural network, the neural network comprising: a speaker independent layer applied to the audio mixture of the plurality of speakers to generate a speaker independent output common to all of the plurality of speakers; a speaker bias layer applied independently once to the speaker-independent output for each of the plurality of speakers to generate a speaker-biased output for each of the plurality of speakers, each application of the speaker bias layer being individually assigned to a corresponding speaker by inputting the corresponding time-invariant speaker embedding, the method further comprising: extracting data indicative of a time-frequency activity region for each of the plurality of speakers in the audio mixture from the combined speaker bias outputs; and rendering the extracted data.

2. 2. The method of claim 1 , wherein the time-invariant speaker embedding is constant across runs of the deep neural network.

3. dividing the audio mixture into a sequence of audio segments; 2. The method of claim 1 , further comprising: running the deep neural network on the sequence of audio segments, wherein the time-invariant speaker embedding is shared between processing of different audio segments of the sequence of audio segments.

4. 2. The method of claim 1 , wherein rendering the extracted data includes outputting a time-frequency mask, the time-frequency mask including an estimate of the time-frequency activity region for each speaker of the plurality of speakers to which a nonlinear function is applied.

5. combining the outputted time-frequency mask with the audio mixture; The method of claim 4 , further comprising generating an output for a speaker from the plurality of speakers based on the combination.

6. The method of claim 5 , wherein the single speaker output comprises a text output indicative of speech transcription data for the single speaker.

7. 10. The method of claim 1, wherein the deep neural network is trained in a weakly supervised training process that includes training the deep neural network based on training data that includes temporal annotation data associated with the audio mixture.

8. 8. The method of claim 7, wherein the deep neural network is trained with the temporal annotation data including ground truth data, the ground truth data including data about diarization information and data about ground truth separated sound sources, such that a diarization loss is calculated based on weak labels and a separation loss is calculated based on strong labels, and the deep neural network is trained using a loss obtained by combining the diarization loss and the separation loss.

9. The method of claim 1 , wherein the time-invariant speaker embeddings comprise speaker embedding vectors obtained based on audio segments of speech forming the audio mixture when only one speaker is active.

10. The method of claim 1 , wherein the deep neural network includes a joint estimation layer for extracting the data indicative of time-frequency activity regions for each of the multiple speakers.

11. 1. A voice processing system, comprising: a memory for storing instructions; a processor for executing the stored instructions to perform the steps of a method, the method comprising: receiving an audio mixture formed by one or a combination of simultaneous and continuous speech of a plurality of speakers and identification information in the form of a time-invariant speaker embedding for each of the plurality of speakers; and processing the audio mixture with a deep neural network, the deep neural network comprising: (1) a speaker independent layer applied to the audio mixture of the multiple speakers to generate a speaker independent output common to all of the multiple speakers; (2) a speaker bias layer applied independently once to the speaker-independent output for each of the plurality of speakers to generate a speaker-biased output for each of the plurality of speakers, each application of the speaker bias layer being individually assigned to a corresponding speaker by inputting the corresponding time-invariant speaker embedding, the method further comprising: extracting data indicative of a time-frequency activity region for each of the plurality of speakers in the audio mixture from the combined speaker bias outputs; and rendering the extracted data.

12. 12. The speech processing system of claim 11, wherein the time-invariant speaker embedding is constant across runs of the deep neural network.

13. The method further comprises: dividing the audio mixture into a sequence of audio segments; and executing the deep neural network on the sequence of audio segments, wherein the time-invariant speaker embedding is shared between processing of different audio segments of the sequence of audio segments.

14. 12. The speech processing system of claim 11, wherein rendering the extracted data includes outputting a time-frequency mask, the time-frequency mask including an estimate of the time-frequency activity region for each speaker of the plurality of speakers to which a nonlinear function is applied.

15. The method further comprises: combining the outputted time-frequency mask with the audio mixture; and generating an output for a single speaker from the plurality of speakers based on the combination.

16. 16. The speech processing system of claim 15, wherein the output of the single speaker comprises a text output indicative of speech transcription data of the single speaker.

17. 12. The audio processing system of claim 11, wherein the deep neural network is trained in a weakly supervised training process that includes training the deep neural network based on temporal annotation data associated with the audio mixture.

18. 18. The speech processing system of claim 17, wherein the deep neural network is trained with the temporal annotation data including ground truth data, the ground truth data including data on diarization information and data on ground truth separated sound sources, such that a diarization loss is calculated based on weak labels and a separation loss is calculated based on strong labels, and the deep neural network is trained using a loss obtained by combining the diarization loss and the separation loss.

19. 12. The speech processing system of claim 11, wherein the time-invariant speaker embeddings comprise speaker embedding vectors obtained based on audio segments of speech forming the audio mixture when only one speaker is active.

20. 12. The speech processing system of claim 11, wherein the deep neural network includes a joint estimation layer for extracting the data indicative of time-frequency activity regions for each of the multiple speakers.

Images