LOW LATENCY SPEAKER SEPARATION

Abstract

A speech separation system comprises data processing hardware and storage hardware that communicates with the data processing hardware, wherein the storage hardware stores instructions which, when executed on the data processing hardware, cause the data processing hardware to perform operations that include: (i) generating a two-dimensional representation of a speech mixture, (ii) separating the speech mixture into an initial separation, (iii) feeding the initial separation and speaker representations to a refinement module, (iv) refining the initial separation based on the initial separation and speaker representations, (v) estimating one mask per speaker, and (vi) applying the masks to the two-dimensional representation to generate two-dimensional speaker-specific representations.

Description

INTRODUCTION

[0001] The information in this section is intended to provide a general context for the disclosure. Work by the presently named inventors, to the extent described in this section, as well as aspects of the description that may not be prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0002] The present disclosure relates generally to a system and method for speech separation, and more particularly to a system and method for low latency, single microphone speech separation using speaker representations.

[0003] Speech separation aims to separate the target speech from background noise and can be used in a variety of applications. Speech separation can be used, for example, by a processor linked to a hearing aid or by a communication device such as a mobile phone or tablet. In each of the aforementioned applications, speech separation is used to separate the desired speech from background noise such as other speakers, noise, and / or music.

[0004] In the context of a hearing prosthesis, such as a hearing aid, a processor connected to the hearing aid separates desired signals associated with speakers, music, and / or a communication device, such as a mobile phone or tablet, from background noise, such as signals associated with construction work, street noise, wind, and the like. In the context of a communication device, such as a mobile phone, a processor connected to the mobile phone separates desired signals associated with a speaker from undesired signals associated with interference, such as signals associated with background music, construction noise, street noise, and the like.

[0005] When a phone call is made in a vehicle via a mobile device or a system linked to the vehicle, a processor linked to the mobile device or the vehicle attempts to separate desired signals associated with the speech of the person speaking or persons in the vehicle cabin from undesired signals associated with background noise in and around the vehicle. As can be imagined, the noise level in a vehicle is often very high and caused by a variety of sources. Background noise can come from road noise, construction work, the operation of vehicle systems such as radios, windows, and door locks, to name a few, the operation of other vehicles, as well as other vehicle occupants.

[0006] In the case of background noise originating from other vehicle occupants, speech separation can be used to separate the speech of the various occupants in the interior and assign the speech to the corresponding occupant. In this context, speech separation can be used to solve the so-called cocktail party problem, in which several people are speaking simultaneously in one place. In this example, the multiple vehicle occupants are all in the same vehicle interior and can speak simultaneously, so it is necessary to separate the speech and assign it to the individual occupants.

[0007] In short, while voice separation is a difficult task for any device, voice separation in a vehicle is often made more difficult due to the constantly changing environment in the vehicle, the numerous conditions in and around the vehicle that contribute to interference, and the general unpredictability of interference during vehicle operation.

[0008] The documents CN 1 16 665 678 A and CN 1 13 990 344 A disclose generating a two-dimensional representation of a speech mixture and using this and speaker representations to generate speaker-dependent masks. SUMMARY

[0009] A system for speech separation is provided and comprises computing hardware and memory hardware in communication with the computing hardware, the memory hardware storing instructions that, when executed on the computing hardware, cause the computing hardware to perform operations that include: (i) generating a two-dimensional representation of a mixture of speech, (ii) separating the mixture of speech into an initial separation, (iii) feeding the initial separation and speaker representations to a refinement module, (iv) refining the initial separation based on the initial separation and the speaker representations, (v) estimating a mask per speaker, and (vi) applying the masks to the two-dimensional representation to generate two-dimensional speaker-specific representations.

[0010] The system may include one or more of the following optional features. For example, generating a two-dimensional representation of the mixture may involve passing the mixture through an encoder. Additionally or alternatively, at least one feature projection may be provided to the refinement module for use by the refinement module in refining the initial separation.

[0011] A speaker embedding table can be used to refine the speaker representations. Additionally or alternatively, the two-dimensional speaker-related representations can be passed through a decoder to generate speaker-related waveforms.

[0012] A microphone can be connected to the data processing hardware and configured to capture the mixed voice. The microphone can be installed in a vehicle.

[0013] In another configuration, a system for speech separation is provided and includes computing hardware and memory hardware in communication with the computing hardware, the memory hardware storing instructions that, when executed on the computing hardware, cause the computing hardware to perform operations including: (i) generating a two-dimensional representation of a mixture of speech, (ii) separating the mixture of speech into an initial separation, (iii) feeding the initial separation and at least one feature projection to a refinement module, (iv) refining the initial separation based on the initial separation and the at least one feature projection, (v) estimating a mask per speaker, and (vi) applying the masks to the two-dimensional representation to generate two-dimensional speaker-specific representations.

[0014] The system may include one or more of the following optional features. For example, generating a two-dimensional representation of the mixed speech may involve passing the mixed speech through an encoder. Additionally or alternatively, speaker representations may be fed to the refinement module for use by the refinement module in refining the initial separation.

[0015] A speaker embedding table can be used to refine the speaker representations. Additionally or alternatively, the two-dimensional speaker-related representations can be passed through a decoder to generate speaker-related waveforms.

[0016] A microphone can be connected to the data processing hardware and configured to capture the mixed voice. The microphone can be installed in a vehicle.

[0017] In a further configuration, a system for speech separation is provided and includes computing hardware and memory hardware in communication with the computing hardware, the memory hardware storing instructions that, when executed on the computing hardware, cause the computing hardware to perform operations including: (i) generating a two-dimensional representation of a mixture of speech, (ii) separating the mixture of speech into an initial separation, (iii) providing the initial separation to a speaker module, (iv) generating frame-specific representations of the initial separation consistent with stored speaker representations, (v) refining the initial separation based on the initial separation and the speaker representations, (vi) estimating a mask per speaker, and (vii) applying the masks to the two-dimensional representation,to create two-dimensional speaker-specific representations.,

[0018] The system may include one or more of the following optional features. For example, generating a two-dimensional representation of the mixed speech may involve passing the mixed speech through an encoder. Additionally or alternatively, a speaker embedding table may be used to refine the speaker representations.

[0019] In one configuration, the two-dimensional speaker-specific representations can be passed through a decoder to generate speaker-specific waveforms.

[0020] A microphone can be connected to the data processing hardware and configured to capture the mixed voice. The microphone can be installed in a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Fig. 1 is a perspective view of a vehicle according to the principles of the present disclosure; Fig. 2 is a schematic diagram of a body control module for use in the vehicle of Fig. 1; and Fig. 3 is a flowchart illustrating a system and method for speech separation according to the principles of the present disclosure.

[0022] Corresponding reference numbers indicate corresponding parts in the drawings. DETAILED DESCRIPTION

[0023] Example configurations will now be described in more detail with reference to the accompanying drawings. Example configurations are provided so that this disclosure will be thorough and will convey the full scope of the disclosure to those skilled in the art. Specific details are set forth, such as examples of specific components, devices, and methods, in order to provide a thorough understanding of configurations of the present disclosure. Those skilled in the art will appreciate that specific details need not be used, that example configurations may be implemented in many different forms, and that the specific details and example configurations should not be construed to limit the scope of the disclosure.

[0024] The terminology used herein is for the purpose of describing specific example configurations only and is not intended to be limiting. As used herein, the singular articles "a," "an," and "the" can also include the plural forms unless the context clearly indicates otherwise. The terms "comprises," "comprising," "containing," and "having" are inclusive and therefore specify the presence of features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein should not be construed as necessarily being performed in the order discussed or illustrated unless they are expressly identified as being in order of performance.Additional or alternative steps may be applied.

[0025] When an element or layer is described as being "on," "engaging," "connected," "attached to," or "coupled" to another element or layer, it may be directly on, engaging, connected, attached, or coupled to the other element or layer, or there may be intervening elements or layers. Conversely, when an element is described as being "directly on," "directly engaging," "directly connected to," "directly attached to," or "directly coupled to" another element or layer, there may be no intervening elements or layers. Other words used to describe the relationship between elements should be interpreted similarly (e.g., "between" versus "directly between," "adjacent" versus "directly next to," etc.).As used herein, the term “and / or” includes all combinations of one or more of the related listed items.

[0026] The terms "first," "second," "third," etc., may be used herein to describe various elements, components, regions, layers, and / or sections. These elements, components, regions, layers, and / or sections should not be limited by these terms. These terms may only be used to distinguish one element, component, region, layer, or section from another region, layer, or section, respectively. Terms such as "first," "second," and other numerical terms do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer, or section discussed below could be referred to as a second element, component, region, layer, or section without departing from the teachings of the example configurations.

[0027] In this application, including the definitions below, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include: an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor (shared, dedicated, or group) executing code; memory (shared, dedicated, or group) storing code executed by a processor; other suitable hardware components providing the described functionality; or a combination of some or all of the above, e.g., in a system-on-chip.

[0028] The term "code" as used above can include software, firmware and / or microcode and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" includes a single processor that executes some or all of the code from multiple modules. The term "group processor" includes a processor that, in combination with other processors, executes some or all of the code from one or more modules. The term "shared memory" includes a single memory that stores some or all of the code from multiple modules. The term "group memory" includes memory that, in combination with other memory, stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium".The term "computer-readable medium" does not encompass transitory electrical and electromagnetic signals propagating through a medium and can therefore be considered tangible and non-transitory storage. Non-limiting examples of non-transitory storage include tangible, computer-readable medium, including non-volatile memory, magnetic storage, and optical storage.

[0029] The devices and methods described in this application may be implemented partially or entirely by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on at least one non-transitory, tangible, computer-readable medium. The computer programs may also include and / or access stored data.

[0030] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0031] Non-transitory memory can be physical devices used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), phase-change memory (PCM), and floppy disks or tapes.

[0032] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a procedural and / or object-oriented high-level language and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) designed to deliver machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to convey machine instructions and / or data to a programmable processor.

[0033] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuits, integrated circuits, purpose-built ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be used for special or general purpose purposes and is coupled to receive data and instructions from and to transmit data and instructions to a storage system, and at least one input device and at least one output device.

[0034] The processes and logic flows described in this specification may be performed by one or more programmable processors, also known as data processing hardware, which run one or more computer programs to perform functions by responding to input data and generating output. The processes and logic flows may also be performed by special-purpose logic circuits, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Processors suitable for executing a computer program include, for example, both general-purpose and special-purpose microprocessors, as well as one or more processors of any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random-access memory, or both.The essential elements of a computer are a processor for executing instructions, and one or more storage devices for storing instructions and data. Generally, a computer will also include, or be operatively connected to, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks, or to receive or transfer data to or from them. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data includes all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable media; magneto-optical disks; and CD-ROM and DVD-ROM disks.The processor and memory can be supplemented by or integrated into special logic circuits.

[0035] To enable interaction with a user, one or more aspects of the disclosure may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or a touchscreen for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, with which the user can provide input to the computer. Other types of devices may also be used to interact with the user; for example, the feedback to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including auditory, voice, or tactile input.In addition, a computer can interact with a user by sending and receiving documents to and from a device used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0036] In Fig. 1, a vehicle 10 is shown. The vehicle 10 includes an interior 12, one or more seats 14 arranged within the interior 12, and a microphone 16. As shown in Fig. 1, the microphone 16 is located within the interior 12 near one or both seats 14. For example, the microphone 16 may be disposed between two front seats 14 such that it is substantially the same distance from each seat 14. In one configuration, the microphone 16 is located in a headliner 20 of the vehicle 10 to place the microphone 16 in close proximity to occupants of the vehicle 10 when they are seated in the seats 14. Although the vehicle 10 is described and illustrated with a single microphone 16, the vehicle 10 could include multiple microphones 16 located at various locations within the interior 12.

[0037] In Fig. 2 schematically shows a body control module (BCM) 20 of the vehicle 10. The BCM 20 includes data processing hardware 22 and storage hardware 24, which cooperate to Fig. 3. The data processing hardware 22 and the storage hardware 24 include an encoder 26 in communication with the microphone 16 and a decoder 28 in communication with an external device 30, such as a mobile phone.

[0038] The encoder 26 may use a fast Fourier transform (FFT) to process a speech or composite waveform signal 32 received from the microphone 16. The FFT may be a short-time fast Fourier transform (STFT) and used in conjunction with a learned filter bank. For example, the encoder 26 may use a learned filter bank with a frame size of 16 samples and an overlap of eight (8) samples with a depth of 512, implemented with a one-dimensional convolutional layer followed by a ReLU (rectified linear unit) activation function. The encoder 26 may use the FFT to convert the composite speech signal 32 from its original time domain into a two-dimensional representation of the composite speech signal 32.For example, the encoder 26 may transform the original time domain of the composite speech signal 32 into the frequency domain to generate a two-dimensional representation of the signal 32.

[0039] Decoder 28 may similarly use an FFT technique to convert the two-dimensional representation of composite speech signal 32 into a waveform signal (i.e., return composite speech signal 32 from the frequency domain to the time domain) before transmitting the signal to external device 30. Decoder 28 may use an inverse short-term FFT (iSTFT) and may also be used in conjunction with a learned filter bank. For example, encoder 26 may use a learned filter bank with a frame size of 16 samples and an overlap of eight (8) samples with a depth of 512, implemented with a one-dimensional convolutional layer followed by a ReLU (rectified linear unit) activation function.It should be noted that the decoder 28 does not convert the same composite speech signal 32 received from the encoder 26 into the time domain, but rather a two-dimensional representation of a separate speech signal generated based on the time domain representation shown in . Fig. 3, as described in more detail below.

[0040] As in Fig. 2, the storage hardware 24 stores, and the data processing hardware 22 executes, an early separation (ES) module 34, a separating refinement (SR) module 36, a mask module 38, a speaker module 40, a feature projection (FP) module 42, and an embedding table 44. As described in more detail below, the ES module 34, the SR module 36, and the speaker module 40 cooperate to separate the two-dimensional representation of the composite speech 32 received from the encoder 26 into individual speakers, which is then sent to the decoder 28 for use by the decoder 28 in generating speaker-specific waveforms. Thus, the ES module 34, the SR module 36, and the speaker module 40 together form a speech separator 47.

[0041] With particular reference to Fig. 3, a flowchart detailing the operation of the speech module 40 and the associated encoder 26 and decoder 28 is described. The speech module 40 is a masking-based encoder-decoder model. As will be described, the input speech or waveform mixture 32 captured by the microphone 16 is processed by the encoder 26, which generates a two-dimensional representation 46 of the mixture 32. The two-dimensional representation 46 is then passed to the masking module 38, which estimates a mask per speaker. The masks are applied to the two-dimensional signal representation 46 to create two-dimensional estimates 48 of the two-dimensional speaker-specific signal representation 46. The estimates 48 are then passed through the decoder 28 to obtain the speaker-specific waveforms 50.

[0042] In operation, the input composite speech signal 32 can be captured by the microphone 16 when one or more occupants are seated in the interior 12. For example, the microphone 16 can capture speech from occupants in the respective front seats 14 of the interior 12. The captured speech includes a mixture of the two seated occupants and additional background noise. The input composite speech signal 32 is transmitted to the encoder 26, which converts the composite speech signal 32 from its original time domain into a two-dimensional representation 46 of the composite speech signal 32, as described above.

[0043] The two-dimensional representation 46 is transmitted to the ES module 34, which creates estimates of speaker masks. The estimated speaker masks 52 are applied to the two-dimensional representation 46 to obtain coded speaker estimates or an initial separation 54 of the two-dimensional representation 46. The coded speaker estimates 54 are then sent to the SR module 36.

[0044] The ES module 34 is initially trained as a standalone separator within an encoder-decoder model. The ES module 34 can be trained by utterance-level permutation invariant training (uPIT) with the same reconstruction loss. For example, the ES module 34 can be trained as a separator with uPIT, using a negative scale-invariant signal-to-distortion ratio (SI-SDR) as the loss term. The same encoder 26 and decoder 28 can also be used for the Fig. 3 can be used, in which the speech separator 47 is used as an early separation module. When training the Fig. 3, the ES module 34 and the encoder 26 are frozen while the training of the decoder 28 continues. The Fig. The overall system shown in Figure 3 can then be trained using the negative SI-SDR as the loss. Note that although the ES module 34 and the encoder 26 are described as frozen after training, the ES module 34, together with the encoder 26, is Fig. 3 could be further trained to continuously improve the ability of the ES module 34 to separate the two-dimensional representation 46.

[0045] A temporal convolutional network (TCN) can be used to implement the ES module 34. The TCN can have 512 convolutional block channels, a skip connection path, and a residual (bottleneck). With 128 channels, each extended convolution has a kernel size of three (3). The ES module 34 is constructed from three (3) iterations with eight (8) convolutional blocks per iteration. In one configuration, all convolutions are implemented as causal, resulting in a fully causal model. Alternatively, the lower convolutional blocks of the ES model 34 can be non-causal to enable a small look-ahead filter.

[0046] The encoded speaker estimates 54 are sent to the speaker module 40. The speaker module 40 receives the speaker estimates or the initial separation 54 and generates frame-wise representations 56 that are consistent with a stored representation of the speaker. These frame-wise representations can be used to separate the speakers in the two-dimensional representation 46. Consistency with the speaker representation is trained using the embedding table 44, as described in more detail below.

[0047] The average value of one hundred (100) frames with 50% speaker representation overlap is compared to the speaker embedding table values ​​in embedding table 44. If the speaker representations 56 deviate from the table values ​​for the speakers contained in embedding table 44, the loss term is increased. In other words, the more the speaker representations 56 deviate from the values ​​contained in embedding table 44 for a particular speaker, the higher the loss term.

[0048] For example, a classification loss is used to measure speaker variance, where the loss increases as the representation estimate for a particular speaker (A) is distant from the stored representation for speaker (A) in the embedding table 44. The loss may also be increased when the representation estimate of speaker (A) is close to another stored representation in the embedding table 44 (i.e., when the representation estimate for speaker (A) is close to the stored representation for speaker (B)). The losses may be used to train the speaker module 40 so that the representation estimate for each speaker approaches its table value and moves away from the table value of another speaker.In this way, the speaker module 40 is trained to bring the speaker representations 56 closer to those contained in the embedding table 44, whereby the speaker module 40 generates more accurate speaker representations 56 in the future.

[0049] A causal TCN with ten (10) convolutional blocks and no repetition can be used in the embedding table 44. The causal TCN can have a kernel of three (3), 512 channels, and 128 bottleneck channels. The first layer of the speaker module 40 can receive all originally separate signals together, so that for each frame, the module 40 receives a size vector of the complete filter bank representation multiplied by the number of speakers. For example, with two (2) speakers and a representation size of 512, the speaker module 40 receives 1024 size vectors per frame. The speaker module 40 outputs a vector of the number of estimated speakers according to the size of the speaker representation. In one configuration, the size of the speaker representation is 512, so for two (2) speakers, the module 40 outputs two (2) vectors of size 512 per frame, which are concatenated into a vector of length 1024.

[0050] As in Fig. 3, the embedding table 44 receives the speaker representations 56 and assists in training the speaker module 40. The embedding table 44 is determined by the number of speakers in the dataset and the speaker representation size. For example, for a dataset of one hundred (100) speakers with a speaker representation (embedding) size of 512, the table 44 would have the following dimensions: 100 x 512. The embedding table 44 is used only in the training phase and is adjusted to pull each speaker representation toward the representation generated by the speaker module 40 to achieve better separation refinement results. While the embedding table 44 is used only in the training phase, the embedding table 44 could be used in an ongoing manner to continually improve the ability of the speaker module 40 to determine the speaker representations 56.

[0051] The speaker representations 56 generated by the speaker module 40 are transmitted to the feature projection module 42. The concatenated speaker representations are projected onto the SR module 36 as element-wise addition and multiplication features on a bottleneck layer of the SR module 36 by transferring the features through fully connected (FC) layers from the concatenated speaker representation vector to a vector the size of the bottleneck layer of the SR module 36. In one configuration, for 24 bottleneck representations in the SR module 36, 48 FC layers could be used, each receiving an input of size 1024 and outputting a vector of length 128. For each bottleneck representation, one of the vectors is used to element-wise multiply the representation, while the other is added to the result.

[0052] As shown, the SR module 36 receives three (3) sets of signals. The SR module 36 receives the original two-dimensional representation 46 of the mixture 32, the initial separation 54 provided by the ES module 34, and the speaker representations 56 via the feature projection 42. The first layer of the SR module 36 receives all of the originally separated signals concatenated. For each frame, the SR module 36 receives a magnitude vector for the complete filter bank representation multiplied by the number of speakers. The SR module 36 also receives the encoded mixture representation 54. In the case where the system and method are fully causal, the initially separated signals are concatenated to form the encoded mixture representation 54 and fed together to the SR module 36. If a lookahead buffer is implemented, causal alignment may also be used.

[0053] The SR module 36 provides a mask for each estimated speaker. The masks provided by the SR module 36 are applied to the original coded mixture 54 and decoded back into the waveform domain by the decoder 28 into separate channels. The SR module 36 may include a TCN with the same dimensional properties as the ES module 34, except for handling the above-mentioned additional inputs and using causal alignment.

[0054] While the speaker module 40 is temporally aligned with the initially separated signals 54, the initially separated signals 54 are not aligned when using lookahead (ie, the mixture representation 54). In such a situation, causal alignment can be used to align the initially separated signals 54.

[0055] Lookahead in an otherwise causal model can be achieved by converting several layers to non-causal ones. In a lookahead implementation, the same selection of non-causal layers is used in both separators for processing the mixture representation 54. The early-separated signals are processed only using causal layers, in parallel with the processing of the original mixture 46 by lookahead layers. When the non-causal part of the SR module 36 is complete, the representation is summed with the representation of the early-separated signals 54.

[0056] In one configuration, the SR module 36 begins processing the early separation signals 54 using separate convolution blocks for the same number of non-causal layers used for mixture processing. As non-causal blocks pass through the SR module 36, the products of the mixed and early-separated signals are summed, achieving timing alignment between the processing of the mixture and the early separation.

[0057] As described, the system and method of Fig.3 shows the initial separation results of a composite speech 32, speaker representations 56, along with feature projections 42, and the initial composite speech 32, provided by the ES module 34 to separate the initial composite speech 32. The SR module 36 uses the aforementioned information to generate masks in a masking module 38. The generated masks are applied to the original two-dimensional representation 46 of the composite speech 32 to generate separated representations 48. The separated representations 48 are sent to a decoder 28, which converts them into individual speaker waveforms or speech estimates 50. The method described above provides improved speech separation performance and low latency.

[0058] A number of implementations have been described. However, it should be understood that various changes may be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are also within the scope of the following claims.

[0059] The foregoing description is for purposes of illustration and description. It is not intended to be exhaustive or limiting of the disclosure. Individual elements or features of a particular configuration are generally not limited to that particular configuration, but are optionally interchangeable and may be used in a selected configuration even if not specifically shown or described. The same may also be varied in many respects. Such variations are not to be considered as departing from the disclosure, and all such changes are intended to be included within the scope of the disclosure.

Claims

[1] Language separation system comprising: Data processing hardware (22); Memory hardware (24) in communication with the data processing hardware (22), the memory hardware (24) storing instructions that, when executed on the data processing hardware (22), cause the data processing hardware (22) to perform operations comprising: Generating a two-dimensional representation (46) of a speech mixture (32); Separating the language mixture (32) into an initial separation (54); Providing the initial separation (54) and speaker representations (56) to a refinement module (36); Refining the initial separation (54) based on the initial separation (54) and the speaker representations (56); Estimating one mask per speaker; and Applying the masks to the two-dimensional representation (46) to generate two-dimensional speaker-specific representations. [2] The system of claim 1, wherein generating a two-dimensional representation (46) of the composite speech (32) includes passing the composite speech (32) through an encoder (26). [3] The system of claim 2, wherein the encoder (26) uses a Fast Fourier Transform (FFT) method to process the composite speech (32). [4] The system of claim 3, wherein the FFT is a fast short-time Fourier transform (STFT) used in conjunction with a learned filter bank. [5] The system of claim 1, further comprising providing at least one feature projection to the refinement module (36) for use by the refinement module (36) in refining the initial separation (54). [6] The system of claim 1, further comprising using a speaker embedding table (44) to refine the speaker representations (56). [7] The system of claim 1, further comprising passing the two-dimensional speaker-specific representations through a decoder (28) to generate speaker-specific waveforms (50). [8] The system of claim 7, wherein the decoder (28) uses an FFT method to convert the two-dimensional speaker-specific representations into the speaker-specific waveforms (50). [9] The system of claim 1, further comprising a microphone (16) in communication with the data processing hardware (22) and configured to capture the mixed voice (32). [10] A vehicle having the system according to claim 9.

Images